{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2af0b51",
   "metadata": {},
   "source": [
    "# CV in Transformer\n",
    "- **Learner** : shenhao\n",
    "- **Date** : 2021.10.14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c9fa6de-50cd-4ffb-9bc9-9b4e7878bbce",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math, copy, time\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "94bcdfff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Using {} device'.format(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc28cadf",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "与其他seq2seq模型类似，我们使用学习到的embedding将输入token和输出token转换为$d_{\\text{model}}$维的向量。我们还使用普通的线性变换和softmax函数将解码器输出转换为预测的下一个token的概率 在我们的模型中，两个嵌入层之间和pre-softmax线性变换共享相同的权重矩阵，类似于[(cite)](https://arxiv.org/abs/1608.05859)。在embedding层中，我们将这些权重乘以$\\sqrt{d_{\\text{model}}}$。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1a27198e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Embeddings(nn.Module):\n",
    "    def __init__(self, d_model, vocab):\n",
    "        \"\"\"\n",
    "        类的初始化函数\n",
    "        d_model：指词嵌入的维度\n",
    "        vocab:指词表的大小\n",
    "        \"\"\"\n",
    "        super(Embeddings, self).__init__()\n",
    "        # 之后就是调用nn中的预定义层Embedding，获得一个词嵌入对象self.lut\n",
    "        self.lut = nn.Embedding(vocab, d_model)\n",
    "        # 最后就是将d_model传入类中\n",
    "        self.d_model = d_model\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Embedding层的前向传播逻辑\n",
    "        参数x：这里代表输入给模型的单词文本通过词表映射后的one-hot向量\n",
    "        将x传给self.lut并与根号下self.d_model相乘作为结果返回\n",
    "        \"\"\"\n",
    "        embedds = self.lut(x)\n",
    "        return embedds * math.sqrt(self.d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "15c52f3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embeddings(\n",
      "  (lut): Embedding(10, 16)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# embedding_size=16, input_vocab_size=10\n",
    "embedding_fc = Embeddings(16, 10).to(device)\n",
    "print(embedding_fc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "18996f82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[4, 2, 2, 7, 4]]), torch.Size([1, 5]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sentence or batch_size=1,words=5\n",
    "input_X = torch.randint(0, 10, (1, 5))\n",
    "input_X, input_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ebc1c0f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 5, 16])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeds_x = embedding_fc(input_X)\n",
    "embeds_x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ccb7b04",
   "metadata": {},
   "source": [
    "## Positional Encodding\n",
    "Positional Encodding位置编码的作用是为模型提供当前时间步的前后出现顺序的信息.因为Transformer不像RNN那样的循环结构有前后不同时间步输入间天然的先后顺序,所有的时间步是同时输入,并行推理的,因此在时间步的特征中融合进位置编码的信息是合理的.\n",
    "\n",
    "位置编码可以有很多选择,可以是固定的,也可以设置成可学习的参数.\n",
    "\n",
    "这里,我们使用固定的位置编码.具体地,使用不同频率的sin和cos函数来进行位置编码,如下所示:\n",
    "$$PE_{pos,2i}=sin(pos/10000^{2i/d_{model}})$$  \n",
    "$$PE_{pos,2i+1}=cos(pos/10000^{2i/d_{model}})$$  \n",
    "\n",
    "其中pos代表时间步的下标索引,向量$PE_{pos}$也就是第pos个时间步的位置编码,编码长度同Embedding层."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1b346024",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, d_model, dropout, max_len=5000):\n",
    "        \"\"\"\n",
    "        位置编码器类的初始化函数\n",
    "\n",
    "        共有三个参数，分别是\n",
    "        d_model：词嵌入维度\n",
    "        dropout: dropout触发比率\n",
    "        max_len：每个句子的最大长度\n",
    "        \"\"\"\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "\n",
    "        # Compute the positional encodings\n",
    "        # 注意下面代码的计算方式与公式中给出的是不同的，但是是等价的，你可以尝试简单推导证明一下。\n",
    "        # 这样计算是为了避免中间的数值计算结果超出float的范围，\n",
    "        pe = torch.zeros(max_len, d_model)\n",
    "        position = torch.arange(0, max_len).unsqueeze(1)\n",
    "        div_term = torch.exp(torch.arange(0, d_model, 2) *\n",
    "                             -(math.log(10000.0) / d_model))\n",
    "        pe[:, 0::2] = torch.sin(position * div_term)\n",
    "        pe[:, 1::2] = torch.cos(position * div_term)\n",
    "        pe = pe.unsqueeze(0)\n",
    "        self.register_buffer('pe', pe)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.pe[:, :x.size(1)].requires_grad_(False)\n",
    "        return self.dropout(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f04a6113",
   "metadata": {},
   "outputs": [],
   "source": [
    "# embedding_size=15,dropout=0.1\n",
    "position_encoding_layer = PositionalEncoding(16, 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "47988e53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 5, 16])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "position_enc_x = position_encoding_layer(embeds_x)\n",
    "position_enc_x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c2e533",
   "metadata": {},
   "source": [
    "> 可以用代码，简单看下效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b8945fd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(69.0, 0.5, 'sequence length')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAJcCAYAAADpbuAhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABYrElEQVR4nO3deZxdVZX3/883CQHCDCEhkGgAA4pIIgZEEJlppBm0W2lp8YlAmwaFBhQRmqdVekRAkP5pQ6cDDSqioCCIDEnzCDRqwhATCGMAI4aEhHkKZKr1++OcijeVW6dupWrvW7fq+/Z1X3XvmdaqkqR29tlnLUUEZmZmZlbfoGYnYGZmZtaXebBkZmZmVsGDJTMzM7MKHiyZmZmZVfBgyczMzKyCB0tmZmZmFTxYMktA0mclTWti/HdJelPS4E72f1PSDxu81lWS/rl3M+w9km6TNKnZeZhZ/+XBktk6kvRRSb+R9JqklyX9WtIeABFxTUQc2qzcIuLZiNg4IlaljCPp85JWlQOz9td3E8Zba5AXER+PiKtTxTQzG9LsBMxakaRNgVuAk4HrgKHAvsCyZubVJL+NiI82Owkzs1Q8s2S2bnYCiIhrI2JVRLwdEdMi4iFYPeNyb/vBkkLSSZLmSXpF0vckqdy3xmyJpLHl8UNqrvWMpDck/V7SZ8vtgyT9X0l/kLRE0vclbdbJNbaXdHd5jenA8NpvRtL1kp4vZ8nukfT+nvxwOn7/NT+D95Tvryp/Br8sc5opaceaY98vaXo5Y7dY0t9LOgz4e+CvyhmsOeWxd0n6m278TCZJelbSi5LO7cn3aWYDgwdLZuvmSWCVpKslfVzSFg2ccwSwBzAeOAb4s65OkLQR8O/AxyNiE2BvYHa5+/Pl6wBgB2BjoLNbYD8CHqQYJP0T0HGNz23AOGAEMAu4poHvp6eOBc4DtgCeAv4FQNImwP8AtwPbAu8B7oyI24F/BX5S3mIcX+ean6frn8lHgZ2Bg4CvS3pfr35XZtbveLBktg4i4nWKX7oB/BfwgqSbJY2sOO38iHg1Ip4FfgVMaDBcG7CrpA0jYlFEPFJu/yxwcUQ8ExFvAucAn2mfTWon6V0Ug7R/iIhlEXEP8IsO38+VEfFGRCwDvgmMb5+RacBekl6tee3V4Hk3RMR9EbGSYnA2odx+BPB8RHw7It4p85rZ4DUb+ZmcV84EzgHmUAxezcw65cGS2TqKiMci4vMRMRrYlWIW5DsVpzxf834pxaxHVzHeAv4KOAlYVN62em+5e1vgDzWH/4FiHWLHAdu2wCvltWqPBUDSYEnnS3pa0uvA/HLXGrfqKsyIiM1rXjMaPK+zn8cY4OkGr9FRIz+Tbv//YGYDmwdLZr0gIh4HrqIYNHXXW8Cwms/bdLj2HRFxCDAKeJxiJgtgIfDumkPfBawEFne4/iJgi/KWXu2x7f4aOBo4GNgMGFtuV3e/kRprfE+Stqk4tqM/Ajt2si+6OLfRn4mZWcM8WDJbB5LeK+krkkaXn8dQrMFpdFal1mzgYypqI21GceuoPc5ISUeVA51lwJtAezmAa4EzysXbG/On9Twray8eEX8AHgDOkzRU0keBI2sO2aS89ksUA5x/XYfvoaM5wPslTZC0AcWtvUbdAmwj6XRJ60vaRNKHy32LgbGSOvu7q6GfiZlZd3iwZLZu3gA+DMyU9BbFIGku8JXuXigipgM/AR6iWIR9S83uQeU1FwIvA/sBXyz3XQn8ALgH+D3wDnBqJ2H+usz3ZeAbwPdr9n2f4nbVc8CjrNuAr+P39CTwjxQLtecB91afsca5bwCHUAzoni/PP6DcfX359SVJs+qc3p2fiZlZQxTR1ay2mZmZ2cDlmSUzMzOzCh4smZmZWZ8i6cqysOzcTvZL0r9LekrSQ5J2r9l3mKQnyn1n90Y+HiyZmZlZX3MVcFjF/o9TFNIdB0wGLoOiFArwvXL/LsCxknbpaTIeLJmZmVmfUhbPfbnikKOB70dhBrC5pFHAnsBTZWHa5cCPy2N7pCmNdMseT5cCg4GpEXF+1fH3b/dJr0I3M7MBY4/nbuxJnbNuW/HiM9l+zw7dese/pZgNajclIqZ08zLbUdRka7eg3FZv+4fpoeyDpZopskMovon7Jd0cEY/mzsXMzMzyKgdG3R0cdVRvMBkV23ukGbfhkkyRmZmZ2YCxgKI1UrvRFPXoOtveI80YLHU2dbYGSZMlPSDpgRvfmp8rNzMzs4GnbVW+V++4Gfg/5VNxewGvRcQi4H5gXFnFfyjwmfLYHmnGmqWGpshqp+m8ZsnMzGzgkHQtsD8wXNICis4D6wFExOXArcDhwFMUDbGPL/etlHQKcAfFuugrI+KRnubTjMFSt6fIPvCL45MmVGvQ1u/u+qBe9OyfnZU13rvvuSxrvIc/eEbWeB/43SVZ482Z8OWs8cbPvjhrvP78/fXn7w1g9vhud97pkQlzvp01Xn///rKLtmZnsIaIOLaL/QF8qZN9t1IMpnpNM27DJZkiMzMzM0sh+8xSqikyMzMzW0dtfWtmqa9pSp2lFFNkZmZmZik0ZbBkZmZmfUf0sTVLfY3bnZiZmZlVaEYF7zHA94FtgDaKMueXVp6zyfAcqQHQ9tzj2WIBbDJiWdZ4uSlrwX4zM1snXrNUqRm34VYCX4mIWZI2AR6UNN3tTszMzKwvasbTcIuAReX7NyQ9RlHB24MlMzOzZvCapUpNXbMkaSzwQWBmnX2r251M/fFN2XMzMzMzgyY+DSdpY+BnwOkR8XrH/bXtTpbN+43bnZiZmVlTNGWwJGk9ioHSNRFxQ1fHLz72H9InVRo+ebdssQDWH9m/V0BLHueamfV5vdfgtl/KfhtOkoArgMciIm+zJDMzM7NuasbM0j7A54CHJc0ut/19WdXbzMzMcvMC70rNeBruXqB/33syMzOzfsPtTszMzAY6F6Ws1BKDpSMXvJ0t1t2/mp0tFsDgkRtljZd7Ed/gQV7gbX3T+NkXM2fCl5udhpm1gGaWDhgMPAA8FxFHNCsPMxuYPFAy+xM30q3WzKKUpwGPNTG+mZmZWZeaMliSNBr4c2BqM+KbmZlZjba2fK8W1KyZpe8AZwGd/tRq2528tHRxtsTMzMzMajWjKOURwJKIeLDquIiYEhETI2LiVsNGZsrOzMxsAIq2fK8W1KyilEdJOhzYANhU0g8j4rjOTnjyteeyJffE3R/KFgtg1y8OyxovN7c7MWsO/9kz6z3NKEp5DnAOgKT9gTOrBkpmZmaWmHvDVWrm03BmZmZmfV5Ti1JGxF3AXc3MwczMbMBr0bVEuXhmyczMzKxCS7Q7OX/rfbPF+n+ZW/x+YKst8gbM/K+HQV5kamZmLa4pgyVJm1MUpNwVCOCEiPhtM3IxMzMb8Fq0WGQuzZpZuhS4PSI+JWko0L+fnzczM7OWlX2wJGlT4GPA5wEiYjmwPHceZmZmVvIC70rNWOC9A/AC8N+SfidpqqSNOh5U2+7kN2/Oy5+lmZmZGc0ZLA0Bdgcui4gPAm8BZ3c8qLbdyd4bj8udo5mZ2cDhRrqVmrFmaQGwICJmlp9/Sp3BUq3JF+2UPKl2x5x5X7ZYAKdvvmPWeLn/Q9UgPw1nZmatrRntTp6X9EdJO0fEE8BBwKO58zAzM7NChNudVGnW03CnAteUT8I9AxzfpDzMzMzMKjVlsBQRs4GJzYhtZmZmHfhpuEpud2JmZmZWoVkVvM8A/oaievfDwPER8U5nxw/Z/9hcqfHo0p9niwXApgdnDRe5250M7t8LvN3Oxcz6hRZ9Si2X7DNLkrYD/g6YGBG7AoOBz+TOw8zMzKwRzVrgPQTYUNIKilYnC5uUh5mZmXnNUqXsM0sR8RxwEfAssAh4LSKmdTyutoL31O//OHeaZmZmZkBzesNtARwNbA+8Clwv6biI+GHtcRExBZgCsGLJPC8MMTMzS6XNdZaqNONpuIOB30fECxGxArgB2LsJeZiZmZl1qRlrlp4F9pI0DHibooL3A1UnLLvgzBx5AbDwrZeyxQLQpltljZe93YmfFjMzsxbXjHYnMyX9FJgFrAR+R3m7zczMzJrAC7wrNauC9zeAbzQjtpmZmVl3NKt0gJmZmfUVLkpZye1OzMzMzCokm1mSdCVwBLCkrNSNpC2BnwBjgfnAMRHxSlfX+sL1SpXmWlasWpktFgAbbZY3Xu52J17gbWbW93nNUqWUM0tXAYd12HY2cGdEjAPuLD+bmZmZ9VnJZpYi4h5JYztsPhrYv3x/NXAX8LVUOZiZmVkDvGapUu41SyMjYhFA+XVEZwfWtjt56s35ufIzMzMzW0OfXeAdEVMiYmJETHzPxmObnY6ZmVn/1daW79WCcpcOWCxpVEQskjQKWNLISdcvuj9xWmvaYMjQrPE0LPMi78w9gAYN9iJvMzNrXbkHSzcDk4Dzy683ZY7fJQ+UepcHSmZmfV+EG+lWSXYbTtK1wG+BnSUtkHQixSDpEEnzgEPKz2ZmZmZ9Vsqn4Y7tZNdBqWKamZnZOmjRtUS59NkF3mZmZmZ9gXvDmZmZDXSu4F0pd7uTC4EjgeXA08DxEfFqV9c6ZtSeqdJcy6y3F2SLBaANNsoaj8ztXDTIC7zNzKx7JB0GXAoMBqZGxPkd9n8V+Gz5cQjwPmDriHhZ0nzgDWAVsDIiJvY0n9ztTqYDu0bEbsCTwDkJ45uZmVmLkTQY+B7wcWAX4FhJu9QeExEXRsSEiJhAMZa4OyJerjnkgHJ/jwdKkLndSURMq/k4A/hUqvhmZmbWoL61wHtP4KmIeAZA0o8p2qU92snxxwLXpkyomQu8TwBu62yn252YmZn1P7W/38vX5A6HbAf8sebzgnJbvWsNo7iL9bOazQFMk/RgnWuvk6Ys8JZ0LrASuKazYyJiCjAF4K/f/UkvfDEzM0sl4wLv2t/vnVC90zo59kjg1x1uwe0TEQsljQCmS3o8Iu5Zx3SBJgyWJE2iWPh9UEQ0NAiackzanGp9/rpR+YIBDN0wa7h4+42s8VTvP3kzM7POLQDG1HweDSzs5NjP0OEWXEQsLL8ukXQjxW29Hg2Wst6GK1e3fw04KiKW5oxtZmZmnehbjXTvB8ZJ2l7SUIoB0c0dD5K0GbAfNa3TJG0kaZP298ChwNye/nhSlg64FtgfGC5pAfANihXr61NMiwHMiIiTUuVgZmZmrSUiVko6BbiDonTAlRHxiKSTyv2Xl4d+EpgWEW/VnD4SuLEcYwwBfhQRt/c0p9ztTq5IFc/MzMzWUR8rShkRtwK3dth2eYfPV1GUKard9gwwvrfzcbsTMzMzswpud2JmZjbQ9a06S31O1nYnNfvOBC6kKE3+YlfXWv+rF6RJso59fnJetlgAGjI0a7xwuxMzM7Nuyd3uBEljgEOAZxPGNjMzs0b1rafh+pxkg6WyANTLdXZdApxF5wWmzMzMzPqM3HWWjgKei4g5DRy7uhz61O//OEN2ZmZmA1S05Xu1oGwLvMv+LedSFIjqUm059BVL5nkWyszMzJoi59NwOwLbA3PKYlGjgVmS9oyI56tOXPmrH2VIr3CAXs8WC4BBg/PGW7Uia7hBgz3ONTPr81p0LVEu2QZLEfEwMKL9s6T5wMRGnoYzMzMza5Zka5bKdie/BXaWtEDSialimZmZmaWSu91J7f6xqWKbmZlZN7Towutc3O7EzMzMrILbnZiZmQ10XuBdKXu7E0mnAqcAK4FfRsRZXV3r8q/OS5XmWk485O1ssZrC7U56VfFgp5mZ9WcpZ5auAr4LfL99g6QDgKOB3SJimaQRnZxrZmZmuXjNUqXc7U5OBs6PiGXlMUtSxTczMzPrDbkXeO8E7CtppqS7Je3R2YG17U5++2a+23BmZmYDjhvpVso9WBoCbAHsBXwVuE6qv+ojIqZExMSImPiRjcflzNHMzMxstdxPwy0AboiIAO6T1AYMB16oOumcJffkyA2Akz7219liAdC2Kmu4yNzuRC5OYWbW97XojE8uuX+V/Rw4EEDSTsBQwO1OzMzMrM9KWTrgWmB/YLikBcA3gCuBKyXNBZYDk8pZJjMzM2sW/yqu1Ix2J8elimlmZmbW21zB28zMbKDzmqVKXn5rZmZmViFruxNJE4DLgQ0o2p18MSLu6+paO28+OlWaa9Gue2WLBWRvP5K93Yl8H9zMrM/zzFKllDNLVwGHddh2AXBeREwAvl5+NjMzM+uzUi7wvkfS2I6bgU3L95sBC1PFNzMzswa5N1yl3Au8TwfukHQRxazW3p0dKGkyMBlgu012YKthI7MkaGZmZlYr9wLvk4EzImIMcAZwRWcH1rY78UDJzMzMmiX3zNIk4LTy/fXA1EZOumXMhskS6mjQqLx96GLl8qzxyNzuZNAQL/A2M+vzvMC7Uu6ZpYXAfuX7A4F5meObmZmZdUvudidfAC6VNAR4h3JNkpmZmTWR251Uaka7kw+limlmZmbW29zuxMzMbKDzmqVKLTFYGvGjf8oWS0PzLSYHaHvl+azxslfwdkMdMzNrccl+lUkaI+lXkh6T9Iik08rtW0qaLmle+XWLVDmYmZlZA9ra8r1aUMp/968EvhIR7wP2Ar4kaRfgbODOiBgH3Fl+NjMzM+uTUi7wXgQsKt+/IekxYDvgaIqn5ACuBu4CvpYqDzMzM+uC251UyrKipOwR90FgJjCyHEi1D6hGdHLOZEkPSHpg6o9vypGmmZmZ2VqSL/CWtDHwM+D0iHhdUkPnRcQUYArAsnm/cQEIMzOzRKLNv2arJB0sSVqPYqB0TUTcUG5eLGlURCySNApY0tV14o0XU6a5Zqz1hmaLBcDbb+SNtyJzexX5D6CZmbW2lE/DiaJR7mMRcXHNrpspesRRfvU9NjMzs2by03CVUs4s7QN8DnhY0uxy298D5wPXSToReBb4dMIczMzMzHok5dNw9wKdLVA6KFVcMzMz6yY/DVfJ9ZXNzMzMKrREu5M5R1yVLdb4n3wyWywAMi8oj1UrssZzuxMzM2t1zWh3cqGkxyU9JOlGSZunysHMzMwa0Bb5Xi2oGe1OpgO7RsRuwJPAOQlzMDMzM+uR7O1OImJazWEzgE+lysHMzMwa0KKP9OfSjHYntU4AbuvknNXtTn6+9PeJMzQzMzOrL3u7k5rt51Lcqrum3nm17U5mbvsXrXmT08zMrBV4ZqlSM9qdIGkScARwUER0ORD69Dvz0iXZweOzf5stFoDet1vWeLkNaonnLc3MzDqX7FdZZ+1OJB0GfA3YLyKWpopvZmZmDep63mJAa0a7k38H1gemF+MpZkTESQnzMDMzM1tnzWh3cmuqmGZmZrYOvGapkusrm5mZmVVIuWZpDPB9YBugDZgSEZfW7D8TuBDYOiJerLrW0pXLUqW5lnf+55FssQA23G501nhsuHHeeP18OC75Pr+Z9QMtWlk7l5RrltoreM+StAnwoKTpEfFoOZA6BHg2YXwzMzOzHkv27/6IWBQRs8r3bwCPAduVuy8BzgI8lDUzM2u2aMv3akHZK3hLOgp4LiLmdHHO6gre7yx/LUeaZmZm1gdIOkzSE5KeknR2nf37S3pN0uzy9fVGz10XWSt4U9yaOxc4tKvzait4D990J89AmZmZpdKH1ixJGgx8j2K5zgLgfkk3R8SjHQ7934g4Yh3P7ZakM0t1KnjvCGwPzJE0HxgNzJK0Tco8zMzMrGXsCTwVEc9ExHLgx8DRGc7tVNYK3hHxMDCi5pj5wMSunoa7YtjEVGmuZe7MFdliAexx1MtZ4zF0g6zh1M+fhjMzs+6RNBmYXLNpSnk3qd12wB9rPi8APlznUh+RNAdYCJwZEY9049xuyV7BOyJclNLMzKwPiYxFKWuX2XSiXkHrjvcJZwHvjog3JR0O/BwY1+C53daMCt61x4xNFd/MzMxa0gJgTM3n0RSzR6tFxOs172+V9B+Shjdy7rpwT3gzM7OBrg8t8AbuB8ZJ2h54DvgM8Ne1B5RrnRdHREjak2IN9kvAq12duy48WDIzM7M+IyJWSjoFuAMYDFwZEY9IOqncfznwKeBkSSuBt4HPREQAdc/taU5NaXci6VTgFIpSAr+MiLOqrnXotQemSnMt/3b8r7LFApj4ct4F3tpyeNZ4/b3diZlZv9DHikWW65tv7bDt8pr33wW+2+i5PZW93QkwkuIxvt0iYpmkEZVXMTMzM2uilAu8FwGLyvdvSGpvd/IF4PyIWFbuW5IqBzMzM2tA31qz1Odkb3cC7ATsK2mmpLsl7dHJOavbnVxxyz050jQzMzNbS9Z2JxHxuqQhwBbAXsAewHWSdigXZq1WW4fh7V9N9ZDXzMwslYx1llpR0sFSnXYnUNRAuKEcHN0nqQ0YDrzQ2XUGj+tx8c2GzVz1s2yxANpe3DBrvMHvXp41nvy8pfVR42dfzJwJX252GmbWArK2Oyn9HDgQuEvSTsBQoLLdiZlZb/NAyayG1yxVyt7uBLgSuFLSXGA5MKnjLTgzMzOzvqJZ7U6OSxXXzMzMuqmP1Vnqa1wy0MzMzKyCl9+amZkNdF6zVCl7uxNJE4DLgQ0oqnx/MSLuq7rWm6eekirNtTy+9JVssQBWLclbwHzwihVZ42lQZ3dizczMWkMz2p1cAJwXEbdJOrz8vH/CPMzMzMzWWTPanQSwaXnYZsDCVDmYmZlZ18JFKSs1o93J6cCFkv4IXASc08k5q9udXD1/UY40zczMzNaSfLDUsd0JcDJwRkSMAc6gKFy5loiYEhETI2LipLGjUqdpZmY2cLVFvlcLaka7k0nAaeX764GpXV3nL+8fmibBOha/9Wq2WAArnt8ia7yhy5dljefiFGZm1uqa0e5kIbAfcBdF25N5qXIwMzOzBrTojE8uzWh38gXgUklDgHeAyQlzMDMzM+uRZrU7+VCquGZmZtZNbndSyStKzMzMzCq43YmZmdlA5zVLlVIu8N4AuAdYv4zz04j4hqQtgZ8AY4H5wDERUdlj5N4lj6ZKcy25/3NZ+kLe8eowtzsxMzPrlpS34ZYBB0bEeGACcJikvYCzgTsjYhxwZ/nZzMzMmiTaIturFSUbLEXhzfLjeuUrgKOBq8vtVwOfSJWDmZmZWU+lLko5GHgQeA/wvYiYKWlk2TeOiFgkaUQn506mLCswaPBmDBq0UcpUzczMBq4WnfHJJenTcBGxKiImAKOBPSXt2o1zV7c78UDJzMzMmiXL6uKIeFXSXcBhwGJJo8pZpVHAkq7OP2Pbj6VOcbX/fPG+bLEAXntpWNZ4wzMv8GaIF3ibmfV5ba6zVCXZzJKkrSVtXr7fEDgYeBy4maI/HOXXm1LlYGZmZtZTKWeWRgFXl+uWBgHXRcQtkn4LXCfpROBZ4NMJczAzMzPrkZTtTh4CPlhn+0vAQanimpmZWTd5gXcltzsxMzMzq+B2J2ZmZgOdZ5YqNaPdyYXAkcBy4Gng+Ih4tepa3/zKFqnSXMut/7x1tlgALy3dIGu8HZctyxqvv7c7GST/BWNm1t81o93JdGDXiNgNeBI4J2EOZmZm1oWIyPZqRdnbnUTEtIhYWW6fQVGw0szMzKxPSrrAW9JgSbMpCk9Oj4iZHQ45Abitk3MnS3pA0gNX/vrRlGmamZkNbG2R79WCmtbuRNK5wErgmk7OXd3u5IR9dkmZppmZmVmnmtHuZK6kScARwEHRwA3M9Y49M3GGfzLhonyxAF5YPjRrPFas7PqY3uTiFGZmfV+Lzvjkkr3diaTDgK8BR0XE0lTxzczMzHpDM9qdPEVRTmC6JIAZEXFSwjzMzMysQnhmqVIz2p28J1VMMzMzs97mCt5mZmYDnWeWKnn5rZmZmVmF7O1OavafCVwIbB0RL1Zda8X3/y1VmmvZd9WwbLEAXsg8txfLlucN2M/bnZiZ9QttzU6gb0v5q7q93cmbktYD7pV0W0TMkDQGOAR4NmF8MzMzsx7L3u6k/HwJcFbNZzMzM7M+KXu7E0lHAc9FxJwuzv1Tu5PfPp4yTTMzswEt2iLbqxUlXTETEauACWVxyhsl7QacCxzawLlTgCkASy/+Qmv+dM3MzKzl5W53cjSwPTCnLEg5Gpglac+IeL6z88/99zdypAnA3w57PVssgOuWbZE1HitWZA0nL/A2M+v7WnTGJ5fc7U5+FxEjImJsRIwFFgC7Vw2UzMzMzJope7uThPHMzMxsXbh0QKXs7U46HDM2VXwzMzOz3uB2J2ZmZgNcqz6llktLDJa+u/B/s8X6x8njs8UCePGWlVnjxTt5F3gzxB11zMystaVc4L2BpPskzZH0iKTzavadKumJcvsFqXIwMzOzBrRlfLWg7O1OgA0pSgjsFhHLJI1ImIOZmZlZj6Rc4B1AvXYnJwPnR8Sy8rglqXIwMzOzrnnNUrXs7U6AnYB9Jc2UdLekPTo5d3W7k7a2t1KmaWZmZtap3O1Odi1jbgHsBewBXCdph3Imqvbc1e1O1hu6nYe8ZmZmqbToWqJccrc7OYyiavcN5eDoPkltwHDghc7O32/krjnSBGDIPnUnupJ5+RcPZ40XyzK3O5HbnZiZWfdIOgy4FBgMTI2I8zvs/yzwtfLjm8DJETGn3DcfeANYBayMiIk9zSfZYEnS1sCKcqDU3u7kWxTf1IHAXZJ2AoYCL6bKw8zMzKpFH5pZKjt/fA84hGKC5X5JN0fEozWH/R7YLyJekfRxijtRH67Zf0BE9NrYInu7E0lDgSslzQWWA5M63oIzMzOzAWtP4KmIeAZA0o8pnqJfPViKiN/UHD8DGJ0yoeztTiJiOXBcqrhmZmbWd0maDEyu2TSlXKfcbjvgjzWfF7DmrFFHJwK31XwOYJqkAP6zw7XXSUtU8DYzM7OEMt6Gq32AqxP1FrvWvQMl6QCKwdJHazbvExELyzqO0yU9HhH3rHPCpF2ztAFwD7B+GeenEfENSROAy4ENgJXAFyPivqpr/XTv5anSXIt2qRq89r6X4/6s8Vie+cb0IC/wNjOzblkAjKn5PBpY2PEgSbsBU4GPR8RL7dsjYmH5dYmkGylu6/VosJSyzlJ7Be/xwATgMEl7ARcA50XEBODr5WczMzNrkmjL92rA/cA4SduX65w/A9xce4CkdwE3AJ+LiCdrtm8kaZP298ChwNye/nyaUcE7gE3L7ZtRZ7RoZmZmA1NErJR0CnAHRemAKyPiEUknlfsvp5hs2Qr4j7JETXuJgJEUdR2hGOP8KCJu72lOSdcslU/CPQi8B/heRMyUdDpwh6SLKGa29u7k3NULwC750E58fsdtU6ZqZmY2cPWh0gEAEXErcGuHbZfXvP8b4G/qnPcMML6380na7iQiVpW320YDe5YVvE8GzoiIMcAZwBWdnDslIiZGxEQPlMzMzKxZkg6W2kXEq8BdFBW8J1HcZwS4nmLhlZmZmTVJH1uz1Oc0o4L3QmA/isHTgcC8rq417JLvpkpzLVKW8eNqr616J2u8tmV5v79BGw3NGs/MzKy3NaOC96vApZKGAO+wZmEqMzMzy6xVZ3xyaUYF73uBD6WKa2ZmZtabXMHbzMxsgPPMUrW8C1jMzMzMWkzymaVyzdIDwHMRcYSkLYGfAGOB+cAxEfFK1TVWPfq/qdNcbciEQ7PFAngz8wLveGf9rPHYJPN4vG1V1nByNxcz6w/Cf5lVyfGb7DTgsZrPZwN3RsQ44M7ys5mZmVmflHSwJGk08OcUje7aHQ1cXb6/GvhEyhzMzMysmussVUs9s/Qd4CzWLKQ+MiIWAZRfR9Q7UdJkSQ9IeuCKX+a7DWdmZmZWK9lgSdIRwJKIeHBdzq9td3Lin+/by9mZmZmZNSblAu99gKMkHQ5sAGwq6YfAYkmjImKRpFHAkoQ5mJmZWReizQu8q6QsSnkOcA6ApP2BMyPiOEkXUvSHO7/8elNX17rlc/luwx1174RssQDeXJm53ck762WNN2SIq1OYmVlra0ZRyvOB6ySdCDwLfLoJOZiZmVmpVRde55JlsBQRd1E0ziUiXgIOyhHXzMzMrKfc7sTMzGyACxelrOQFJWZmZmYVmtHu5ELgSGA58DRwfES8WnWNyW/MTJ3makfM3y9bLICl2Rd4b5I1HoP8rxUzs77Oa5aqNaPdyXRg14jYDXiS8ok5MzMzs74oe7uTiJgWESvLjzOA0SlzMDMzs2rRpmyvVtSMdie1TgBuq7ejtt3JshWvJ0rPzMzMrFrT2p1IOhdYCVxTb39tu5P119s0VZpmZmYDXkS+VyvqcoG3pPWBvwTG1h4fEf/Yxal1252UVbwnAUcAB0V0/aPbfP2Nujqk18SsGdliAbyzcnnWeKvyricH+YFL65vGz76YORO+3Ow0zKwFNPI03E3Aa8CDwLJGL1zR7uQw4GvAfhGxtLsJm5n1Bg+UzP6kVdcS5dLIYGl0RBzWizG/C6wPTJcEMCMiTurF65uZmZn1mkYGS7+R9IGIeHhdg3Rod/Kedb2OmZmZ9T7PLFXrdLAk6WEgymOOl/QMxW04AVHWSTIzMzPr16pmlo7IloWZmZlZH9XpYCki/gAg6QcR8bnafZJ+AHyu7okddGx3UrP9TOBCYOuIeLHqGjdsNLaRUL3irWlPZ4sFsLxtZdcH9aJVyzI/neZ2J2ZmfV6rPtKfSyO/Od9f+6Ec/HyoGzE6tjtB0hjgEODZblzHzMzMLLtOB0uSzpH0BrCbpNfL1xvAEopyAl2q1+6kdAlFZW+PZc3MzJrM7U6qdTpYioh/i4hNgAsjYtPytUlEbFXWUGrEd+jQ7kTSURS35OZUnVjb7uRnb85vMJyZmZlZ72qkdMD1knbvsO014A81DXHXUtvupCxKiaRhwLnAoV0FjYgpwBSA373raM9AmZmZJRLRmjM+uTQyWPoPYHfgIYqyAR8A5gBbSTopIqZ1ct5a7U6AHwDbA3PKgpSjgVmS9oyI5ztL4H03f6HBb6fn7vmzH2aLBbBiVd4F5SuXe4G3mZlZdzTym3M+8MGyqe2HgAnAXOBg4ILOToqIcyJidESMBT4D/L+I+MuIGBERY8vtC4DdqwZKZmZmlla05Xu1okYGS++NiEfaP0TEoxSDp2fSpWVmZmbWNzRyG+4JSZcBPy4//xXwpKT1gRWNBKltd9Jh+9iGsjQzM7Nk2rxmqVIjM0ufB54CTgfOAJ4pt60ADkiUl5mZmVmf0OXMUkS8DXy7fHX0Zq9nZGZmZln5abhqXQ6WJO0DfBN4d+3xEbFDIwHqtTuRdCpwCrAS+GVEnFV5jWGbNxKqV9y5Yd7/YNpeyVsVYeWywVnjaVDmp+/MzMx6WSNrlq6guP32ILBqHWK0tzvZFEDSAcDRwG4RsUzSiHW4ppmZmfWSVq2snUsj/+x/LSJui4glEfFS+6uRi3fS7uRk4PyIWAYQEUu6nbWZmZlZJo0Mln4l6UJJH5G0e/urwet/hw7tToCdgH0lzZR0t6Q96p1Y2+5k6k9ubjCcmZmZdVdEvlcrauQ23IfLrxNrtgVwYNVJ9dqd1MTcAtgL2AO4TtIOEWv+CGvbnSx78t4W/fGamZlZq2vkabh1LQ+wVrsTST+kqNp9Qzk4uk9SGzAceKGzCy389HnrmEL33ddpt7v+YcWKvAu8+3u7E8njeDOz/q7L23CSRkq6QtJt5eddJJ3Y1XmdtDs5Dvg55ayUpJ2AocCL6/4tmJmZWU9Em7K9WlEja5auAu4Ati0/P0lRoHJdXQnsIGkuRVXwSR1vwZmZmZn1FY2sWRoeEddJOgcgIlZK6lYJgdp2JxGxHDium3mamZlZIm53Uq2RmaW3JG1FsagbSXsBryXNyszMzKyPaGRm6cvAzcCOkn4NbA18KmlWZmZmlo3bnVRr5Gm4WZL2A3YGBDwRESsaDdCx3YmkCcDlFE/IrQS+GBH3VV3jyOffaDRcj72yPF+sZlixPPPTcIMzxzMzM+tlnQ6WJP1FJ7t2kkRE3NBgjDXanQAXAOdFxG1lWYELgP0bvJaZmZn1Mj9mVa1qZunIin0BdDlYqml38i8Ut/Paz20fOG0GLOw6TTMzM7Pm6HSwFBHH98L1v0PR7mSTmm2nA3dIuohigfne9U6UNBmYDDBqk+3ZckP32zUzM0vBT8NVa+RpuHVS2+6kw66TgTMiYgxwBnBFvfMjYkpETIyIiR4omZmZWbM08jTcuuqs3cmRFOuYAK4HpnZ1oadfX5QsyY5WtXWrhFSPDVLe0fyKlbnbnSQbj5vZAOZWQ73LT8NVS/abrKLdyUJgv/KwA4F5qXIwMzMz66kuZ5YkDQO+ArwrIr4gaRywc0Tcso4xvwBcKmkI8A7luiQzMzNrDj8NV62RmaX/BpYBHyk/LwD+uTtBIuKuiDiifH9vRHwoIsZHxIfrrGkyMzOzAUzSYZKekPSUpLPr7Jekfy/3PyRp90bPXReNDJZ2jIgLgBUAEfE2RXFKMzMz6wfaQtleXSmLWX8P+DiwC3CspF06HPZxYFz5mgxc1o1zu62RwdJySRvyp95wO1LMNJmZmZn1tj2BpyLimYhYDvwYOLrDMUcD34/CDGBzSaMaPLfbGnka7hvA7cAYSddQPOX2+UYuLmk+8AawClgZERMlbQn8BBgLzAeOiYhXqq5z4fB9GwnXK05f/KtssQDWG5zygcS1Lc/+NJwnIc3M+rqcT8PV1lEsTYmIKTWftwP+WPN5AfDhDpepd8x2DZ7bbY30hpsuaRawF8Xtt9Mi4sVuxDigw/FnA3dGxPnlvcSzga91J2kzMzNrTeXAaErFIfVGbh2XoHd2TCPndluXt+EkfZJiVuiX5RNwKyV9ogcxjwauLt9fDfTkWmZmZta/LADG1Hwezdqt0To7ppFzu62RNUvfiIjX2j9ExKsUt+YaEcA0SQ+W024AIyNiUXmtRUDd8tySJkt6QNIDv37TpZjMzMxS6UsLvIH7gXGStpc0lKJW480djrkZ+D/lU3F7Aa+VY4pGzu22RhbM1BtQNbrQZp+IWChpBDBd0uONJlY7TffdMce5AoSZmdkAEBErJZ0C3AEMBq6MiEcknVTuvxy4FTgceApYChxfdW5Pc2pk0POApIspHsUL4FSgodpIEbGw/LpE0o0Uq9QXSxoVEYvKletLurrOiZe8t5FwveLrx8/MFgtgVbRljbe8LXP7EbndiZlZX9fXZiQi4laKAVHttstr3gfwpUbP7alGfpOdCiyneILteoqq23UTrCVpI0mbtL8HDgXmUkyHTSoPmwTc1P20zczMzPJo5Gm4tyieWOuukcCNKhrFDgF+FBG3S7ofuE7SicCzwKfX4dpmZmbWSxpcSzRgNdIbbifgTIq6SKuPj4gDq86LiGeA8XW2vwQc1N1EzczMzJqhkTVL1wOXA1MpikuamZlZP5KzKGUramSwtDIiLkueSYUhH/urbLHGbnx7tlgAzy3tTn3PnlseruBtZmbWHY0Mln4h6YvAjdT0hIuIl7s6sZN2JxcCR1IsGn8aOL6s3WRmZmZNkPe57NbTyGCp/cm1r9ZsC2CHBmN0bHcyHTinrIXwLeAc3O7EzMzM+qhGnobbvjcDRsS0mo8zgE/15vXNzMyse6JuSzVr10hvuGGS/q+kKeXncZKOaPD69dqd1DoBuK2TuKvbnUz9/o8bDGdmZmbWuxq5DfffFBW79y4/L6B4Qu6WBs5dq91JRNwDIOlcYCVwTb0Ta9udrFgyr68VFzUzM+s32vxbtlIjg6UdI+KvJB0LEBFvq6w02ZVO2p3cI2kScARwUFmyvNI7//rlRsL1ignrb5MtFsAry9/IGm9Z5qlWDXK7EzMza22NDJaWS9qQsnWMpB2peSquM2WLk0ER8UZNu5N/lHQYxYLu/SJi6bqnbmZmZr2hzWuWKjUyWPoGcDswRtI1wD7A5xs4r7N2J08B61PclgOYEREnrUPuZmZmZsk18jTcdEmzgL0AAad1KAXQ2XmdtTt5z7okamZmZtYMjfSG+1j5tn1xzS6SaF+obWZmZq3NpQOqNXIbrrYY5QYUi7QfBCob6UL9Ct41+84ELgS27mqm6oQb8y0SPnLlBtliAdw3ZMOs8ZYr84JrtzsxM7MW18htuCNrP0saA1zQjRgdK3i3X+MQ4NluXMfMzMwScLuTausyzbAA2LWHcS8BzqJ8ws7MzMysr2pkzdL/x58GNYOACcCcBq/fXsE7gP+MiCmSjgKei4g5VeWayorfkwF233I3dth4bIMhzczMrDu8ZqlaI2uWHqh5vxK4NiJ+3eD116rgDZxLUXOpUm0F70+/+2jPQJmZmVlTNLJm6ep1vXidCt77AdsD7bNKo4FZkvaMiOfXNY6ZmZmtO69ZqtbIbbiHqb+2SEBExG6dnFe3gndEjKg5Zj4wsaun4W5c9EDV7l71rZ3fny0WwH+8lPfpu2WReao199N3mQ2SJz3NzPq7Rm7D3VZ+/UH59bPAUqCrGae6FbzXJUkzMzNLxzNL1RoZLO0TEfvUfD5b0q8j4h+rTuqsgneHY8Y2EN/MzMysaRq5R7KRpI+2f5C0N7BRupTMzMwsp0DZXq2okZmlE4ErJW1GsXbpNeCEpFmZmZmZ9RGNPA33IDBe0qaAIuK1Ri/eWbsTSacCp1CUIvhlRJxVdZ1jR3240ZA9NuKwvAt2N/vx4KzxlrVlHtXnbncSvvNuZtZduX81tJpGnoYbCfwrsG1EfFzSLsBHIuKKBmOs0e5E0gHA0cBuEbGsrMFkZmZm1ic1smbpKuAOYNvy85PA6T2IeTJwfkQsg6IGUw+uZWZmZj3UhrK9WlEjg6XhEXEd5ZOFEbGS4rZaI9rbnTxYti8B2AnYV9JMSXdL2qPeiZImS3pA0gPz3vx9g+HMzMzMelcjC7zfkrQVZWFKSXtRLPJuRL12J0OALYC9gD2A6yTtEBFrLBaqbXfyuXf/hSv/mZmZWVM0Mlj6MnAzsKOkXwNbA59q5OJ12p3sCSwAbigHR/dJagOGAy+sQ/5mZmbWQ56RqNbI03CzJO0H7EzR4uSJiFjR1XmdtTsB3gQOBO6StBMwFKhsd3LZX+e7xzl4j4nZYgFsft19WeMty/0nYnDep/3MzMx6WyNPw30auD0iHpH0f4HdJf1zRMzq4tS67U4kDaWo2zQXWA5M6ngLzszMzPJx0ZVqjdyG+4eIuL6s4v1nwEXAZUBl8aPO2p1ExHLguHXI1czMzCy7Rp6Ga3/y7c+ByyLiJopbZ2ZmZtYPtEnZXq2okcHSc5L+EzgGuFXS+g2eZ2ZmZtbyGrkNdwxwGHBRRLwqaRTw1UYuXq/diaQJwOXABhTtTr4YEZWrnNf/yoWNhOsVbc8/nS0WwJaanTXestyD+kEeV5uZ9XVeOFytkafhlgI31HxeBCzqRow12p0AFwDnRcRtkg4vP+/fjeuZmZmZZdPIzFJvC2DT8v1mwMIm5GBmZmYlPw1XLfU9knrtTk4HLpT0R4on686pd2Jtu5Op3782cZpmZmZm9aWeWarX7uRTwBkR8TNJxwBXAAd3PLG23cmKF5/x7VQzM7NE2lrzIbVskg6WOml3Mgk4rTzkemBqV9dZecd/J8uxoyEH5y0BtVXmO6Fe4G1WGD/7YuZM+HKz0zCzFpDsN5mkjSRt0v6eot3JXIo1SvuVhx0IzEuVg5lZZzxQMvuTNpTt1YpSTmt01u7kTeBSSUOAd4DJFdcwMzMza6pkg6WKdif3Ah9KFdfMzMy6xwuDq3lBiZmZmVkFD5bMzMzMKiR9FEvS5hRPu+1KMct3AvAE8BNgLDAfOCYiXqm6zv/3979PmeYazviz9bLFAti6Le94dWn2p+FaczGfmdlA4tIB1VL/pr4UuD0i3kuxfukx4GzgzogYB9xZfjYzMzPrk1KWDtgU+BhF0UkiYnlEvAocDVxdHnY18IlUOZiZmVnX2jK+WlHKmaUdgBeA/5b0O0lTy3pLI8tmvO1NeUfUO7m23cnMN12KyczMzJoj5WBpCLA7cFlEfBB4i27ccouIKRExMSImfnjjcalyNDMzG/Ai46sVpVzgvQBYEBEzy88/pRgsLZY0KiIWSRoFLOnqQl9ffE/CNNd02ssLs8UCGL4yazieXi/zf6ryA5dmZtbakv0mi4jngT9K2rncdBDwKHAzRX84yq83pcrBzMzMutamfK9WlLqL66nANZKGAs8Ax1MM0K6TdCLwLPDpxDmYmZmZrbOkg6WImA1MrLProJRxzczMrHGt+pRaLl5QYmZmZlYh9W04MzMz6+M8s1StGe1O/gI4ElgOPA0cXxar7NQuW7wrZZpraPv97GyxAEasWpE13qNDB2eNxyBPXpqZWe+RtCVdtE2TNAb4PrANxVhwSkRcWu77JvAFilqQAH8fEbdWxWxGu5PpwK4RsRvwJHBO4hzMzMysQijfqxc00jZtJfCViHgfsBfwJUm71Oy/JCImlK/KgRI0od1JREyLiPbqQjOA0alyMDMzs36ny7ZpEbEoImaV79+gmKzZbl0DNqPdSa0TgNvqnVzb7uTFpc8nTNPMzGxgy9kbrvb3e/ma3M10G2qb1k7SWOCDwMyazadIekjSlZK26Cpg09qdSDqXYprsmnon17Y7GT5sm4RpmpmZWS61v9/L15SOx0j6H0lz67yO7k4sSRsDPwNOj4jXy82XATsCE4BFwLe7uk4z2p0gaRJwBHBQRHTZf+MXY4cmS7KjtvtnZIsFMHz9d7LGW8awrPE0qEXLtTZIg1q105GZWd8VEQd3tk9SQ23TJK1HMVC6JiJuqLn24ppj/gu4pat8src7kXQY8DXgqIhYmiq+mZmZNSbnbbhe0GXbNEmiWDP9WERc3GHfqJqPnwTmdhWwGe1O7gfWB6YX3wszIuKkxHmYmZlZ/3A+ddqmSdoWmBoRhwP7AJ8DHpY0uzyvvUTABZImUJQ0mg/8bVcBm9Hu5D0pY5qZmVn3tNKCgoh4iTpt0yJiIXB4+f5eoO46kIj4XHdjumKgmZmZWQW3OzEzMxvg2vr3szg9lr3dSUT8ttx3JnAhsHVEvFh1neE//NeUaa7h5Ul5C4pvttl6WeMtf2ODrPGQJy/NzKy1pZ5Zam938qlykfcwWN2z5RCKhVlmZmbWRG6kWy17u5Ny9yXAWbTWmjIzMzMbgLK3O5F0FPBcRMypOrm2HPrUH/88YZpmZmYDW4vVWcou5W249nYnp0bETEmXAt+kmG06tKuTy/LnUwCWPT3DM1BmZmbWFLnbnXwT2B6YUxakHA3MkrRnWfG7rnj5uYRprmnmU6O6PqgX7f3+fN8bwIo3Mo87B3mBt5lZX+cZiWq5253MiogRETE2IsZSDKh2rxoomZmZmTVTM9qdmJmZWR/iOkvVmtHupHb/2JTxzczMzHrKFbzNzMwGuFZ9Si2XlhgszTrqR9lijRD8ZP18VbX332ZwtlgAy57O/EfCC7ytjxo/+2LmTPhytnhzJnyZ8bMvzhbPzHpP0t9kkjaX9FNJj0t6TNJHyu2nSnpC0iOSLkiZQ3flHCiZWfPkHCgBHiiZtbDs7U4kHQAcDewWEcskjUicg5mZmVVw6YBqyQZLNe1OPg9FuxNguaSTgfMjYlm5fUmqHMzMzMx6Knu7E2AnYF9JMyXdLWmPeifXtju5aenvE6ZpZmY2sLUR2V6tKOVgqb3dyWUR8UHgLeDscvsWwF7AV4HrVJbzrhURUyJiYkRMPHrY9gnTNDMzM+tc7nYnZ5fbb4iIAO6T1AYMp5iFquuYZU8kTHNN7x6UdwnVkJEbZ423IjI/Dbf2ONjMzPoYlw6olrvdyaPAz4EDASTtBAwFXkyVh5mZmVlPNKPdyVvAlZLmAsuBSeUsk5mZmTWBfwlXa1a7k+NSxjUzMzPrLS1RwdvMzMzS8Zqlai0xWFq2akW2WE+9uTBbLABt/eGs8ZbzWtZ4bndiZmatLulgSdLmwFRgV4pboicAbwOXAxsAK4EvRsR9KfMwMzOzzrX5weVK2dudANcB50XEbZIOBy4A9k+ch5mZmdk6aUa7kwA2LQ/bDMh738vMzMzW0KqVtXNJObNU2+5kPPAgcBpwOnCHpIso6jztXe9kSZOByQAbbzCCDYZunjBVMzMzs/qa0e7kZOCMiBgDnAFcUe/k2nYnHiiZmZmlExlfragZ7U4+SjHDBHA9xQLwSldtsHuSBOv5xCv3ZIsFoK22zBpvBa9kjYcyPw3X5gdgzcysdzWj3clCYL9y24HAvFQ5mJmZmfVUM9qd3ARcKmkI8A7luiQzMzNrDs/JV2tGu5N7gQ+ljGtmZmbWW1qigreZmZml49IB1VLWWdoZ+EnNph2ArwPfL7ePBeYDx0RE5arjA6//szRJ1nNw5gXem2+RNd6KyDzZ6nYnZmbW4lIu8H4iIiZExASK225LgRspnoi7MyLGAXeWn83MzKxJXDqgWq5/9h8EPB0RfwCOBq4ut18NfCJTDmZmZmbdlmuw9Bng2vL9yIhYBFB+HVHvBEmTJT0g6YErbr4rT5ZmZmYDUFvGVytKvsC7LBtwFHBOd86LiCnAFIC377mqVWfuzMzMrMXleBru48CsiFhcfl4saVRELJI0CliSIQczMzPrhJ+Gq5ZjsHQsf7oFB3AzMAk4v/x6U1cXGLz9B9NkVsfmG26cLRYAm+Vtd7IyVmWN19+fhpOanYGZmaWW9DeZpGHAIcANNZvPBw6RNK/cd37KHMzMzKyan4arlrqC91Jgqw7bXqJ4Os7MzMysz3MFbzMzswGuVZ9Sy6V/LygxMzMz66FmtDvZDjgSWA48DRwfEa9WXeuNL/5doizXtv1G22SLBaCN87Y7WZ57gbdXQJuZ9XnRsquJ8mhGu5PpwK4RsRvwJN2sv2RmZmaWU/Z2JxExLSJWlttnAKMz5WBmZmbWbc1od1LrBOC2eifUtju5+g+LkiZnZmY2kLndSbXkg6WadifXd9h+LrASuKbeeRExJSImRsTESe8elTpNMzMzs7qa0e4ESZOAI4CDIsKryszMzJrI7U6qZW93Iukw4GvAfmXRyi59cla+clATh27V9UG9aaPNsoZb4XYnZmZm3ZJ0FFLT7uRvazZ/F1gfmK7isfIZEXFSyjzMzMysc55XqtaMdifvSRnTzMzMrDe53YmZmdkA5zVL1bygxMzMzKxC9nYnEfGdcv+ZwIXA1hHxYtW1fvvC46nSXMupW+2fLRaAhm2eNd6K3FUuvMDbzKzPa9X6R7kkGyxFxBPABABJg4HnKNqdIGkMxcLvZ1PFNzMzM+sN2dudlJ8vAc7CC/DNzMyaLjL+rxVlb3ci6SjguYiYU3VCbbuTtra3cuRoZmZmfZykLSVNlzSv/LpFJ8fNl/SwpNmSHuju+bWytjsp6y6dC3y9q/Nq250MGrRR6jTNzMwGrBbrDXc2cGdEjAPuLD935oCImBARE9fxfCBzuxNJHwC2B+aUBSlHA7Mk7RkRz3d2gTO33S9DmoW9tu40jSS0/rCs8VZF5mV88gJvMzPrVUcD+5fvrwbuougMkuz8rO1OIuJhYET7DknzgYldPQ1nZmZm6eRcSyRpMjC5ZtOUiJjSjUuMjIhFABGxSNKITo4LYJqkAP6zJkaj56/WjHYnZmZmNkCVg5bKwZGk/wG2qbPr3G6E2iciFpaDoemSHo+Ie7px/mrZ25102D82ZXwzMzNrPRFxcGf7JC2WNKqcFRoFLOnkGgvLr0sk3QjsCdwDNHR+LS8oMTMzG+BabIH3zcCk8v0k4KaOB0jaSNIm7e+BQ4G5jZ7fkQdLZmZm1krOBw6RNI9iqc/5AJK2lXRrecxI4F5Jc4D7gF9GxO1V51dpSrsTSacCpwArKb6Bs6qu9Q9f6/ROXq+LRZmfFltv/azhVsaqrPHc7sTMrO9ri9YpFhkRL1EUu+64fSFwePn+GWB8d86vkr3diaQDKB7b2y0iljWyCt3MzMysWXKUDoCadieSLgTOj4hlUCy8ypSDmZmZ1dE680rNkb3dCbATsK+kmZLulrRHvRNq251cee/ceoeYmZmZJZd8Zqmm3ck5NTG3APYC9gCuk7RDxJo3TGvrMCy97FQPes3MzBJp89xSpaztTsrPC4AbysHRfZLagOHAC51dYL1Pn5E+y9LKGT/PFgtAQ4ZmjZd9gbfbnZiZWYvL8ZtsdbuT0s+BAwEk7QQMBdzuxMzMrEki4/9aUdLBUk27kxtqNl8J7CBpLvBjYFLHW3BmZmZmfUX2dicRsRw4LmVcMzMza1zmCoMtxwtKzMzMzCrkqrNkZmZmfZSfhquWvd0JcBdwObABRbuTL0bEfVXXWvHf/5woy7UNPnJS1wf1pkGDs4ZbFZknW93uxMzMWlz2difAfwHnRcRtkg4HLgD2T5WHmZmZVWvVp9RyyfXP/tXtTiiqqm9abt8MWJgpBzMzM7Nua0a7k9OBCyX9EbiIP1X2XsMa7U5mPJEnSzMzM7MOkg+WatqdXF9uOhk4IyLGAGcAV9Q7LyKmRMTEiJh4wl47p07TzMxswGrL+GpFzWh3Mgk4rXx/PTC1qwuc/b23EqW2tm9P2jZbrGZY0Za53Uk/p0H9+z7/IPXv78/MrBE5Bksd250sBPajeCruQGBehhzMzMysE26kUS3pYKmm3cnf1mz+AnCppCHAO8DklDmYmZmZ9UQz2p3cC3woZVwzMzNrnItSVnPFQDMzM7MKbndiZmY2wLXqU2q5pF6zdAbwNxSFKB8GjgeGUbRBGQvMB46JiFeqrvMfC+9NmeYavp0tUinz02nZ250o7+Rl5P7+zMys30v2m0zSdsDfARMjYldgMEVxyrOBOyNiHHBn+dnMzMyaJDL+rxWl/mf/EGDD8sm3YRRlA44Gri73Xw18InEOZmZmZussZSPd5yRdBDwLvA1Mi4hpkkZGxKLymEWSRtQ7X9JkyrICGrwZgwZtlCpVMzOzAc1Pw1VLeRtuC4pZpO2BbYGNJB3X6Pm17U48UDIzM7NmSbnA+2Dg9xHxAoCkG4C9gcWSRpWzSqOAJV1d6KCRuyVMc01tLy3IFgtg0ObbZI23MlZmjccgV6cwM+vrXMG7WsrfZM8Ce0kaJknAQcBjwM0U/eEov96UMAczMzOzHkm5ZmmmpJ8Cs4CVwO+AKcDGwHWSTqQYUH06VQ5mZmbWNRddqZa63ck3gG902LyMYpbJzMzMrM9zBW8zM7MBrlXrH+XSEoOl6/Zfni1W27wHs8UC0PgDs8bLXsHbC7ytjxo/+2LmTPhys9MwsxaQ9DeZpDMkPSJprqRrJW0g6UJJj0t6SNKNkjZPmYOZWT0eKJlZo5rR7mQ6sGtE7AY8CZyTKgczMzPrWhuR7dWKsrc7iYhpEauL/cwARifOwczMzGydJRssRcRzQHu7k0XAaxExrcNhJwC31Ttf0mRJD0h64Kp5z6VK08zMbMCLiGyvVtS0dieSzqWov3RNvfNr2518ftx2qdI0MzMzq9SMdic/lDQJOAI4KBoYZm74rX9PmOaaVnzvm9liAQx6715Z461qy/w0nPw0nJlZX9eqa4lyyd7uRNJhwNeAoyJiacL4ZmZmZj3WjHYnjwDrA9OLMRQzIuKkVHmYmZlZNRelrNaMdifvSRnTzMzMrDe1RAVvMzMzS6etRZ9Sy6UlBkurHv5Vtlgv3vZKtlgAI495PWu8Nrc7MTMz65bs7U5q9p0pKSQNT5mDmZmZVYuMr1bUjHYnSBoDHELxxJyZmZlZn5W93Um5/RLgLFp3kGlmZtZvuDdcteztTiQdBTwXEXOqzq9td3LFrb9OlaaZmZlZpWQLvDu0O3kVuF7S/wG+BBza1fkRMYWiLhNv3/Hd1hyKmpmZtYBWnfHJJXe7k+MpBk9zyoKUo4FZkvaMiOc7u9DPj/9twjQ7GpkxFnxy6WtZ463K/TSc252YmVmLSzlYWt3uBHibot3JDRFxQPsBkuZTLAB/MWEeZmZmZuusGe1OzMzMrA9poKf9gNaMdie1+8emjG9mZmbWUy1RwdvMzMzS8QLvakkHS5LOAP6Gop7Sw8DxEfGOpFOBUyhuz/0yIs6qus5Jr89ImeYaPjN892yxAD7xRt72Ktn7/7jdiZmZtbiUpQPaK3jvEhFvS7oO+IykP1CUFNgtIpZJGpEqBzMzM+taeGapUjMqeJ8MnB8RywAiYkniHMzMzMzWWfYK3sBOwL6SZkq6W9Ie9c6vreC9fMXrqdI0MzMb8CIi26sVpWykW1vBe1tgI0nHUcw2bQHsBXwVuE5lhcpaETElIiZGxMSh622aKk0zMzOzSrkreO8NLKAoThnAfZLagOHACwlzMTMzs074abhquSt4PwA8BBwI3CVpJ2AoUFnBe6sN8s0szVq+OFssAF7NW7w8e7uTfk7yXzBmZjlJ2hL4CTAWmA8cExGvdDhm5/KYdjsAX4+I70j6JvAF/jRJ8/cRcWtVzGZU8A7gSklzgeXApGjVm5hmZmb9QIv9Gj4buDMizpd0dvn5a7UHRMQTwAQASYOB54Abaw65JCIuajRgsyp4H5cyrpmZmfVbRwP7l++vBu6iw2Cpg4OApyPiD+sa0BUDzczMBrg2Itur9mn38jW5m+mOjIhFAOXXruo1fga4tsO2UyQ9JOnK8oG0Sm53YmZmZtlExBSKZTmdkvQ/wDZ1dp3bnViShgJHAefUbL4M+CeKZUH/BHwbOKHqOtnbnQDvBS4HNqBYy/TFiLiv6jo/33h0yjTXcODLT2SLBRCv5m13kv2+tDx5aWbW1/W1Ct4RcXBn+yQtljQqIhZJGgVUFbf+ODArIlY/vVX7XtJ/Abd0lU/KOkvt7U4mRsSuwGCKqbALgPMiYgLw9fKzmZmZWSNuBiaV7ycBN1UceywdbsGVA6x2nwTmdhWwGe1OAmivBbBZuc3MzMysEecDh0iaBxxSfkbStpJWlwAoSxcdAtzQ4fwLJD0s6SHgAOCMrgKmLB3wnKT2didvA9MiYpqkPwJ3lPsGURSqXEu54GsywD9s+QE+tcm7U6VqZmY2oLW1UOmAiHiJ4gm3jtsXAofXfF4KbFXnuM91N2Yz2p2cDJwREWMoRnNX1Du/tt2JB0pmZmbWLClvw61udxIRKyimwfamuL/YPiV2PbBnwhzMzMysC5Hxf62oGe1OFgL7URSROhCY19WFdr75lHRZdvDaxMqnB3vfK69mDZe93cmgzE/Dtbmdi5mZ9a5mtDv5HXBpuej7Hcp1SWZmZtYcrbRmqRma0e7kXuBDKeOamZmZ9RZX8DYzMxvgWnUtUS4ur2xmZmZWIXW7k9OALwAC/isiviNpS+AnwFhgPnBMRFT3/Bi6Qco01yApWyyAthdezhsv831paXDWeGZm1n1es1QtZZ2lXSkGSnsC44EjJI0DzgbujIhxwJ3lZzMzM7M+KeVtuPcBMyJiaUSsBO6m6MFyNHB1eczVwCcS5mBmZmZdcJ2laikHS3OBj0naqqy1dDgwBhgZEYsAyq8j6p0sabKkByQ9MPW6XyRM08zMzKxzKessPSbpW8B04E1gDkW9pUbPn0JRl4llj9/dmkNRMzOzFuA1S9VS11m6grL3m6R/BRYAiyWNiohFkkYBS7q6zoJP/WvKNNew5YabZIsFsGrxm1njRe4/ELkreJuZmfWypL/JJI0ov74L+AvgWuBmiv5wlF9vSpmDmZmZVfOapWqpi1L+TNJWwArgSxHxiqTzgesknUjRP+7TiXMwMzMzW2epb8PtW2fbSxRNdc3MzMz6PLc7MTMzG+Ai2pqdQp/m1bdmZmZmFZrR7uRC4EhgOfA0cHxEvFp1naNeqNzdq8ZttG22WADLn2+4mkKvaNXFdWZmlk6bfzdUaka7k+nArhGxG/AkcE6qHMzMzMx6Knu7k4iYVn4GmAGMTpiDmZmZdSEisr1aUTPandQ6Abit3sm17U5eebvLupVmZmZmSTSt3Ymkc8vP13Ry/up2J+8f+eHWHIqamZm1AK9ZqtaMdidImgQcARwUDczJzX9jcco013DoiO2zxQJ4c8lLWeO53YmZmVn3pH4abkRELKlpd/IRSYcBXwP2i4ilKeObmZlZ11p1LVEuzWh38l1gfWC6JCgWgZ+UOA8zMzOzddKMdifvSRnTzMzMuqfNM0uVvKDEzMzMrIJ7w5mZmQ1w7u5QLXu7k5p9ZwIXAltHxItV1/n2Vh9NmeYatntnVbZYAK/FhlnjZV/Ep/49eTlI/gvGzKy/SzZY6tDuZDlwu6RfRsQ8SWOAQ4BnU8U3MzOzxvhpuGrZ252U+y4BzgLP+5mZmVnflr3diaSjgOciYk7VybXtTv73zXkJ0zQzMzPrXDPanZwLHNrA+avbnVw+5jjPQJmZmSXidifVcrc7WQx8FphTFqQcDcyStGdEPN/ZdSb9+64p01zDa5fcni0WwCNPjMwaLzu3OzEzsxaXvd1JRFxas38+MLGrp+HMzMwsHS/wrpa93UnieGZmZma9Knu7kw77x6aMb2ZmZl1zu5NqXlBiZmZmVsHtTszMzAY4r1mq1pR2J5JOBU6hKCXwy4g4qzLJvT9ZtbtXbfLoo9liASx5Km97Ff9xMDMz657s7U4oygUcDewWEcskjUiVg5mZmXXNdZaqpZxZWt3uBEBSe7uTicD5EbEMICKWJMzBzMzMrEeytzsBdgL2lTRT0t2S9qh3cm27k6k/+EnCNM3MzAa2iMj2akXNaHcyBNgC2AvYA7hO0g7R4SdY2+5kxeInWvOna2ZmZi0vd7uTBRS3524oB0f3SWoDhgMvdHadd/7pjJRprmHIUUdkiwXw4uDfZI2XnVydwsysr3OdpWrZ250AbcCBwF2SdgKGAm53YmZmZn1S9nYnkq4ErpQ0l+IpuUkdb8GZmZlZPuGn4Splb3cSEcuB41LGNTMzM+stXlBiZmZmVqEl2p1M+kW+NK89bUK2WAAvDP511njZDRqcN1605Y1nLWv87IuZM+HLzU7DrE/wAu9qSWeWJJ0maa6kRySdXm6bIGmGpNllHaU9U+ZgZlaPB0pm1qhmtDu5ADgvIm6TdHj5ef9UeZiZmVk1P2dVrRntTgLYtDxmM2BhwhzMzMzMeiTlYGku8C9l6YC3KdqdPACcDtwh6SKK24B71ztZ0mRgMsCELXdj+43fnTBVMzOzgculA6olW7MUEY8B7e1ObudP7U5OBs6IiDHAGZQVvuucPyUiJkbERA+UzMzMrFma0e7k34DTykOuB6Z2dZ2bFz2YKsW1DNpim2yxAF5kRdZ42bndiZlZn+c1S9VSPw03ovza3u7kWoo1SvuVhxwIzEuZg5mZmVlPNKPdyReASyUNAd6hXJdkZmZmzeGZpWrNaHdyL/ChlHHNzMzMektLVPA2MzOzdDyvVM2rb83MzMyqRES/fQGTHc/x+losx3M8xxs48XJ/b36lefX3maXci8cdr3Xj9efvzfEcz/GaF88PMfUD/X2wZGZmZtYjHiyZmZmZVejvg6Upjud4fTCW4zme4w2ceLm/N0tA5QI0MzMzM6ujv88smZmZmfWIB0tmZmZmFfrlYEnSYZKekPSUpLMzxLtS0hJJczPEGiPpV5Iek/SIpNMSx9tA0n2S5pTxzksZrybuYEm/k3RLhljzJT0sabakBzLE21zSTyU9Xv7/+JGEsXYuv6/21+uSTk8Vr4x5RvnfylxJ10raIGGs08o4j6T6vur9+Za0paTpkuaVX7dIHO/T5ffYJmli4lgXlv9tPiTpRkmbJ473T2Ws2ZKmSdo2ZbyafWdKCknDU8aT9E1Jz9X8GTy8t+JZPv1usCRpMPA94OPALsCxknZJHPYq4LDEMdqtBL4SEe8D9gK+lPj7WwYcGBHjgQnAYZL2Shiv3WnAYxnitDsgIiZERK/9IqpwKXB7RLwXGE/C7zMinii/rwkUPRmXAjemiidpO+DvgIkRsSswGPhMoli7Al8A9qT4OR4haVyCUFex9p/vs4E7I2IccGf5OWW8ucBfAPf0YpzOYk0Hdo2I3YAngXMSx7swInYr/xu9Bfh64nhIGgMcAjzbi7E6jQdc0v7nMCJu7eWYlkG/GyxR/MX5VEQ8ExHLgR8DR6cMGBH3AC+njFETa1FEzCrfv0Hxi3a7hPEiIt4sP65XvpI+FSBpNPDnwNSUcZpB0qbAx4ArACJieUS8min8QcDTEfGHxHGGABtKGgIMAxYmivM+YEZELI2IlcDdwCd7O0gnf76PBq4u318NfCJlvIh4LCKe6K0YXcSaVv48AWYAoxPHe73m40b04t8vFX83XwKc1ZuxuohnLa4/Dpa2A/5Y83kBCQcTzSRpLPBBYGbiOIMlzQaWANMjImk84DsUf5G1JY7TLoBpkh6UlLra7g7AC8B/l7cZp0raKHHMdp8Brk0ZICKeAy6i+Bf7IuC1iJiWKNxc4GOStpI0DDgcGJMoVkcjI2IRFP+AAUZkipvbCcBtqYNI+hdJfwQ+S+/OLNWLdRTwXETMSRmng1PKW41X9uYtW8unPw6WVGdbv6uPIGlj4GfA6R3+ZdbrImJVOUU+GtizvP2RhKQjgCUR8WCqGHXsExG7U9y6/ZKkjyWMNQTYHbgsIj4IvEXv3sKpS9JQ4Cjg+sRxtqCYddke2BbYSNJxKWJFxGPAtyhuG90OzKG4TW29QNK5FD/Pa1LHiohzI2JMGeuUVHHKQfW5JB6QdXAZsCPFMoZFwLczxrZe0h8HSwtY81+Xo0l3G6ApJK1HMVC6JiJuyBW3vF10F2nXZ+0DHCVpPsUt1AMl/TBhPCJiYfl1CcV6nj0ThlsALKiZnfspxeAptY8DsyJiceI4BwO/j4gXImIFcAOwd6pgEXFFROweER+juP0xL1WsDhZLGgVQfl2SKW4WkiYBRwCfjbzF+H4E/GXC6+9IMZCfU/4dMxqYJWmbVAEjYnH5D8424L9I+/eLJdIfB0v3A+MkbV/+a/ozwM1NzqnXSBLFepfHIuLiDPG2bn8aRtKGFL8MH08VLyLOiYjRETGW4v+7/xcRSWYmACRtJGmT9vfAoRS3d5KIiOeBP0raudx0EPBoqng1jiXxLbjSs8BekoaV/60eRMIF7JJGlF/fRbEAOsf3CMXfKZPK95OAmzLFTU7SYcDXgKMiYmmGeLWL8o8i7d8vD0fEiIgYW/4dswDYvfxzmUT7oLr0SRL+/WIJRUS/e1GsXXgSeBo4N0O8aymmV1dQ/OE7MWGsj1LcVnwImF2+Dk8Ybzfgd2W8ucDXM/7/uD9wS+IYO1DcvpkDPJLpv5cJwAPlz/TnwBaJ4w0DXgI2y/T/23kUv/DmAj8A1k8Y638pBptzgIMSxVjrzzewFcVTcPPKr1smjvfJ8v0yYDFwR8JYT1Gs+2z/++XyxN/bz8r/Vh4CfgFslzJeh/3zgeGJv78fAA+X39/NwKgU/536lfbldidmZmZmFfrjbTgzMzOzXuPBkpmZmVkFD5bMzMzMKniwZGZmZlbBgyUzMzOzCh4smWUiaWy97uflvn+UdHCd7ftLuqWTc+b3Zsf0eteV9Jvevn6DOUzN0ADbzKwhQ5qdgJlBRORsv9CwiEhWfbuLuH/TjLhmZvV4Zsksr8GS/kvSI5KmlVXRkXSVpE+V7w+T9LikeymqUlNu36o853eS/pOaPoiSjpN0n6TZkv5T0uBy+5tlk9I5kmZIGtkxoS6u+2b5dX9Jd0u6TtKTks6X9Nky5sOSdiyP21rSzyTdX772Kbd/s2wiepekZyT9Xbl9I0m/LPObK+mvyu13SZpYvj+2jDFX0rdqc+vqezMz6w0eLJnlNQ74XkS8H3iVDn2wJG1A0T/qSGBfoLZn1TeAe6NowHsz8K7ynPcBf0XREHgCsIqiezvARsCMiBgP3AN8oU5Oda9bx3jgNOADwOeAnSJiT2AqcGp5zKXAJRGxR/m9Ta05/73An1H0xvpG2ePwMGBhRIyPiF0pGuLW/jy2pWiWeyBF5fM9JH2iG9+bmVmPebBkltfvI2J2+f5BYGyH/e8tj5kXRXn92ibCH2v/HBG/BF4ptx8EfAi4X9Ls8vMO5b7lQPuap3rxqq7b0f0RsSgillG0EppWbn+45roHA98t87gZ2LS99x7wy4hYFhEvUjSeHVmee7Ckb0naNyJe6xBzD+CuKBrzrqToSv+xbnxvZmY95jVLZnktq3m/CtiwzjFVPYjq7RNwdUScU2ffivhTT6NVdP5nvpG+R7W5t9V8bqu57iDgIxHx9hoJSh3PXwUMiYgnJX2Iop/jv0maFhH/WHtqRT6Nfm9mZj3imSWzvuVxYPv2NUDAsTX77qG8vSbp48AW5fY7gU9JGlHu21LSu7sRs7PrrotpwCntHyRNqDq4vM22NCJ+CFwE7N7hkJnAfpKGl+uwjgXu7kF+Zmbd5sGSWR8SEe8Ak4Fflgu8/1Cz+zzgY5JmAYcCz5bnPAr8X2CapIeA6cCoboSte9119HfAREkPSXoUOKmL4z8A3FfetjsX+OfanRGxCDgH+BUwB5gVETf1ID8zs27Tn2axzczMzKwjzyyZmZmZVfBgyczMzKyCB0tmZmZmFTxYMjMzM6vgwZKZmZlZBQ+WzMzMzCp4sGRmZmZW4f8HGwDhRf8AzJUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 导入依赖库\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import math\n",
    "\n",
    "\n",
    "def get_positional_encoding(max_seq_len, embed_dim):\n",
    "    # 初始化一个positional encoding\n",
    "    # embed_dim: 字嵌入的维度\n",
    "    # max_seq_len: 最大的序列长度\n",
    "    positional_encoding = np.array([\n",
    "        [pos / np.power(10000, 2 * i / embed_dim) for i in range(embed_dim)]\n",
    "        if pos != 0 else np.zeros(embed_dim) for pos in range(max_seq_len)])\n",
    "    positional_encoding[1:, 0::2] = np.sin(\n",
    "        positional_encoding[1:, 0::2])  # dim 2i 偶数\n",
    "    positional_encoding[1:, 1::2] = np.cos(\n",
    "        positional_encoding[1:, 1::2])  # dim 2i+1 奇数\n",
    "    # 归一化, 用位置嵌入的每一行除以它的模长\n",
    "    # denominator = np.sqrt(np.sum(position_enc**2, axis=1, keepdims=True))\n",
    "    # position_enc = position_enc / (denominator + 1e-8)\n",
    "    return positional_encoding\n",
    "\n",
    "positional_encoding = get_positional_encoding(max_seq_len=100, embed_dim=16)\n",
    "plt.figure(figsize=(10, 10))\n",
    "sns.heatmap(positional_encoding)\n",
    "plt.title(\"Sinusoidal Function\")\n",
    "plt.xlabel(\"hidden dimension\")\n",
    "plt.ylabel(\"sequence length\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e4d1a1",
   "metadata": {},
   "source": [
    "## 掩码及其作用\n",
    "掩码：掩代表遮掩，码就是我们张量中的数值，它的尺寸不定，里面一般只有0和1；代表位置被遮掩或者不被遮掩。\n",
    "\n",
    "掩码的作用：在transformer中，掩码主要的作用有两个，一个是屏蔽掉无效的padding区域，一个是屏蔽掉来自“未来”的信息。Encoder中的掩码主要是起到第一个作用，Decoder中的掩码则同时发挥着两种作用。\n",
    "\n",
    "屏蔽掉无效的padding区域：我们训练需要组batch进行，就以机器翻译任务为例，一个batch中不同样本的输入长度很可能是不一样的，此时我们要设置一个最大句子长度，然后对空白区域进行padding填充，而填充的区域无论在Encoder还是Decoder的计算中都是没有意义的，因此需要用mask进行标识，屏蔽掉对应区域的响应。\n",
    "\n",
    "屏蔽掉来自未来的信息：我们已经学习了attention的计算流程，它是会综合所有时间步的计算的，那么在解码的时候，就有可能获取到未来的信息，这是不行的。因此，这种情况也需要我们使用mask进行屏蔽。现在还没介绍到Decoder，如果没完全理解，可以之后再回过头来思考下。\n",
    "\n",
    "mask的构造代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4fdbb9d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def subsequent_mask(size):\n",
    "    # 生成向后遮掩的掩码张量，参数size是掩码张量最后两个维度的大小，它最后两维形成一个方阵\n",
    "    \"Mask out subsequent positions.\"\n",
    "    attn_shape = (1, size, size)\n",
    "\n",
    "    # 然后使用np.ones方法向这个形状中添加1元素，形成上三角阵\n",
    "    subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')\n",
    "\n",
    "    # 最后将numpy类型转化为torch中的tensor，内部做一个1- 的操作。这个其实是做了一个三角阵的反转，subsequent_mask中的每个元素都会被1减。\n",
    "    # 如果是0，subsequent_mask中的该位置由0变成1\n",
    "    # 如果是1，subsequect_mask中的该位置由1变成0\n",
    "    return torch.from_numpy(subsequent_mask) == 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "24f0d191",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0, 1, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 1, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 1, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 1],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]], dtype=uint8)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attn_shape = (1, 10, 10)\n",
    "# np.triu(m, k=0)\n",
    "# k是指从主对角线开始保留\n",
    "# k=0\n",
    "# [[1. 1. 1. 1. 1.]\n",
    "#  [0. 1. 1. 1. 1.]\n",
    "#  [0. 0. 1. 1. 1.]\n",
    "#  [0. 0. 0. 1. 1.]\n",
    "#  [0. 0. 0. 0. 1.]]\n",
    "# k=1\n",
    "# [[0. 1. 1. 1. 1.]\n",
    "#  [0. 0. 1. 1. 1.]\n",
    "#  [0. 0. 0. 1. 1.]\n",
    "#  [0. 0. 0. 0. 1.]\n",
    "#  [0. 0. 0. 0. 0.]]\n",
    "# k=2\n",
    "# [[0. 0. 1. 1. 1.]\n",
    "#  [0. 0. 0. 1. 1.]\n",
    "#  [0. 0. 0. 0. 1.]\n",
    "#  [0. 0. 0. 0. 0.]\n",
    "#  [0. 0. 0. 0. 0.]]\n",
    "subseq_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')\n",
    "subseq_mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "94cf9f86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ True, False, False, False, False, False, False, False, False, False],\n",
       "         [ True,  True, False, False, False, False, False, False, False, False],\n",
       "         [ True,  True,  True, False, False, False, False, False, False, False],\n",
       "         [ True,  True,  True,  True, False, False, False, False, False, False],\n",
       "         [ True,  True,  True,  True,  True, False, False, False, False, False],\n",
       "         [ True,  True,  True,  True,  True,  True, False, False, False, False],\n",
       "         [ True,  True,  True,  True,  True,  True,  True, False, False, False],\n",
       "         [ True,  True,  True,  True,  True,  True,  True,  True, False, False],\n",
       "         [ True,  True,  True,  True,  True,  True,  True,  True,  True, False],\n",
       "         [ True,  True,  True,  True,  True,  True,  True,  True,  True,  True]]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.from_numpy(subseq_mask) == 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "28f4ee22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x272213540a0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATsAAAEvCAYAAAA6m2ZKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAASRElEQVR4nO3cf+xddX3H8edrFf7QkaGlIJQizjQkaCYj33Q65oLzV2mIbGbZ2pjJ1KRjkWQmWzI2E+efc4sucRibOgm6OHSLomQWgRgTNBG1kAJlgFTCQv12VDEDGdtY2Xt/fE+z29tz+/323Hu/fMvn+Uhu7rnn8znnvPu537x6zr33fFJVSNKL3c+90AVI0mow7CQ1wbCT1ATDTlITDDtJTTDsJDXhJS90AX3OesW6unDTaSe93Q/ue+kcqpF0qvgv/oPn6r/T17Ymw+7CTafxvds2nfR27zjvktkXI+mU8d36xsQ2L2MlNWGqsEuyNcnDSQ4kua6nPUk+0bXfl+TSaY4nSUMNDrsk64BPAlcAFwM7klw81u0KYHP32Al8aujxJGka05zZbQEOVNWjVfUc8AXgqrE+VwGfqyV3AWcmOXeKY0rSINOE3Ubg8ZHXB7t1J9tHkuZumrDr+3p3fAqVlfRZ6pjsTLI3yd4fP/n8FGVJ0vGmCbuDwOjvQ84HFgf0AaCqdlfVQlUtbFi/boqyJOl404Td94HNSV6d5HRgO3DLWJ9bgPd038q+AXiqqg5NcUxJGmTwj4qr6kiSa4HbgHXADVX1QJJruvZdwB5gG3AAeBZ47/QlS9LJm+oOiqraw1Kgja7bNbJcwAemOYYkzYJ3UEhqgmEnqQlrciKAoW5b3HfS2zh5gNQGz+wkNcGwk9QEw05SEww7SU0w7CQ1wbCT1ATDTlITDDtJTTDsJDXBsJPUBMNOUhMMO0lNeFFNBDDEkMkDwAkEpFONZ3aSmmDYSWqCYSepCYadpCYYdpKaYNhJaoJhJ6kJhp2kJhh2kpowOOySbEryzSQPJnkgyR/19Lk8yVNJ9nWPD09XriQNM83tYkeAP66qe5KcAdyd5I6q+pexft+qqiunOI4kTW3wmV1VHaqqe7rlnwEPAhtnVZgkzdJMPrNLciHwy8B3e5rfmOTeJLcmee0sjidJJ2vqWU+S/DzwJeCDVfX0WPM9wKuq6pkk24CvAJsn7GcnsBPggo1rfzKWIbOlOFOK9MKZ6swuyWksBd3nq+rL4+1V9XRVPdMt7wFOS3JW376qandVLVTVwob166YpS5KOM823sQE+AzxYVR+f0OeVXT+SbOmO9+TQY0rSUNNcL14G/B5wf5J93bo/By4AqKpdwG8Df5jkCPCfwPaqqimOKUmDDA67qvo2kGX6XA9cP/QYkjQr3kEhqQmGnaQmGHaSmmDYSWqCYSepCYadpCYYdpKaYNhJasLav+P+RWTI5AHgBALSLHhmJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCYSepCYadpCYYdpKaYNhJaoJhJ6kJznpyCnC2FGl6ntlJaoJhJ6kJU4VdkseS3J9kX5K9Pe1J8okkB5Lcl+TSaY4nSUPN4jO7N1fVTya0XQFs7h6/Anyqe5akVTXvy9irgM/VkruAM5OcO+djStJxpg27Am5PcneSnT3tG4HHR14f7NZJ0qqa9jL2sqpaTHI2cEeSh6rqzpH29GxTfTvqwnInwAUb/UWMpNma6syuqha758PAzcCWsS4HgU0jr88HFifsa3dVLVTVwob166YpS5KOMzjskrwsyRlHl4G3A/vHut0CvKf7VvYNwFNVdWhwtZI00DTXi+cANyc5up9/qKqvJ7kGoKp2AXuAbcAB4FngvdOVK0nDDA67qnoUeH3P+l0jywV8YOgxJGlWvINCUhMMO0lN8DceL2JDZktxphS9WHlmJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCYSepCYadpCYYdpKaYNhJaoITAegYQyYPACcQ0NrnmZ2kJhh2kppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCYSepCYadpCYMDrskFyXZN/J4OskHx/pcnuSpkT4fnrpiSRpg8O1iVfUwcAlAknXAj4Cbe7p+q6quHHocSZqFWV3GvgX4YVX964z2J0kzNauw2w7cNKHtjUnuTXJrktfO6HiSdFKmnvUkyenAO4E/62m+B3hVVT2TZBvwFWDzhP3sBHYCXLDRyVhONUNmS3GmFK2mWZzZXQHcU1VPjDdU1dNV9Uy3vAc4LclZfTupqt1VtVBVCxvWr5tBWZL0/2YRdjuYcAmb5JVJ0i1v6Y735AyOKUknZarrxSQvBd4G/MHIumsAqmoX8NvAHyY5AvwnsL2qappjStIQU4VdVT0LrB9bt2tk+Xrg+mmOIUmz4B0Ukppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCYSepCYadpCZ4x71eMEMmDwAnENAwntlJaoJhJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCYSepCYadpCYYdpKaYNhJaoKznuiU42wpGsIzO0lNMOwkNWHZsEtyQ5LDSfaPrHtFkjuSPNI9v3zCtluTPJzkQJLrZlm4JJ2MlZzZ3QhsHVt3HfCNqtoMfKN7fYwk64BPAlcAFwM7klw8VbWSNNCyYVdVdwI/HVt9FfDZbvmzwG/2bLoFOFBVj1bVc8AXuu0kadUN/czunKo6BNA9n93TZyPw+Mjrg906SVp18/yCIj3ramLnZGeSvUn2/vjJ5+dYlqQWDQ27J5KcC9A9H+7pcxDYNPL6fGBx0g6randVLVTVwob16waWJUn9hobdLcDV3fLVwFd7+nwf2Jzk1UlOB7Z320nSqlvJT09uAr4DXJTkYJL3A38JvC3JI8DbutckOS/JHoCqOgJcC9wGPAj8Y1U9MJ9/hiSd2LK3i1XVjglNb+npuwhsG3m9B9gzuDpJmhHvoJDUBMNOUhOc9UTNGDJbijOlvHh4ZiepCYadpCYYdpKaYNhJaoJhJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmGHaSmmDYSWqCEwFIJzBk8gBwAoG1yDM7SU0w7CQ1wbCT1ATDTlITDDtJTTDsJDXBsJPUBMNOUhMMO0lNWDbsktyQ5HCS/SPr/jrJQ0nuS3JzkjMnbPtYkvuT7Euyd4Z1S9JJWcmZ3Y3A1rF1dwCvq6pfAn4A/NkJtn9zVV1SVQvDSpSk6S0bdlV1J/DTsXW3V9WR7uVdwPlzqE2SZmYWn9m9D7h1QlsBtye5O8nOGRxLkgaZataTJB8CjgCfn9DlsqpaTHI2cEeSh7ozxb597QR2Alyw0clYdGobMluKM6XM1+AzuyRXA1cC766q6utTVYvd82HgZmDLpP1V1e6qWqiqhQ3r1w0tS5J6DQq7JFuBPwXeWVXPTujzsiRnHF0G3g7s7+srSfO2kp+e3AR8B7goycEk7weuB85g6dJ0X5JdXd/zkuzpNj0H+HaSe4HvAV+rqq/P5V8hSctY9sOxqtrRs/ozE/ouAtu65UeB109VnSTNiHdQSGqCYSepCYadpCYYdpKaYNhJaoJhJ6kJhp2kJhh2kprgHffSGjFk8gBwAoGV8sxOUhMMO0lNMOwkNcGwk9QEw05SEww7SU0w7CQ1wbCT1ATDTlITDDtJTTDsJDXBsJPUBMNOUhOc9UQ6xTlbysp4ZiepCYadpCYsG3ZJbkhyOMn+kXUfSfKjJPu6x7YJ225N8nCSA0mum2XhknQyVnJmdyOwtWf931TVJd1jz3hjknXAJ4ErgIuBHUkunqZYSRpq2bCrqjuBnw7Y9xbgQFU9WlXPAV8ArhqwH0ma2jSf2V2b5L7uMvflPe0bgcdHXh/s1knSqhsadp8CXgNcAhwCPtbTJz3ratIOk+xMsjfJ3h8/+fzAsiSp36Cwq6onqur5qvpf4NMsXbKOOwhsGnl9PrB4gn3urqqFqlrYsH7dkLIkaaJBYZfk3JGXvwXs7+n2fWBzklcnOR3YDtwy5HiSNK1l76BIchNwOXBWkoPAXwCXJ7mEpcvSx4A/6PqeB/xdVW2rqiNJrgVuA9YBN1TVA/P4R0jScpYNu6ra0bP6MxP6LgLbRl7vAY77WYokrTbvoJDUBMNOUhOc9URq1JDZUk7lmVI8s5PUBMNOUhMMO0lNMOwkNcGwk9QEw05SEww7SU0w7CQ1wbCT1ATDTlITDDtJTTDsJDXBiQAkrdiQyQNgbUwg4JmdpCYYdpKaYNhJaoJhJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmLHsHRZIbgCuBw1X1um7dF4GLui5nAv9eVZf0bPsY8DPgeeBIVS3MpGpJOkkruV3sRuB64HNHV1TV7x5dTvIx4KkTbP/mqvrJ0AIlaRaWDbuqujPJhX1tSQL8DvAbM65LkmZq2s/s3gQ8UVWPTGgv4PYkdyfZOeWxJGmwaWc92QHcdIL2y6pqMcnZwB1JHqqqO/s6dmG4E+CCjU7GIr2YDJktZdYzpQw+s0vyEuBdwBcn9amqxe75MHAzsOUEfXdX1UJVLWxYv25oWZLUa5rL2LcCD1XVwb7GJC9LcsbRZeDtwP4pjidJgy0bdkluAr4DXJTkYJL3d03bGbuETXJekj3dy3OAbye5F/ge8LWq+vrsSpeklVvJt7E7Jqz//Z51i8C2bvlR4PVT1idJM+EdFJKaYNhJaoJhJ6kJhp2kJhh2kppg2ElqgmEnqQmGnaQmeMe9pDVpyOQBW97x7MQ2z+wkNcGwk9QEw05SEww7SU0w7CQ1wbCT1ATDTlITDDtJTTDsJDXBsJPUBMNOUhMMO0lNMOwkNSFV9ULXcJwkPwb+tafpLOAnq1xOH+s4lnUcyzqOtZp1vKqqNvQ1rMmwmyTJ3qpasA7rsA7rOFlexkpqgmEnqQmnWtjtfqEL6FjHsazjWNZxrDVRxyn1mZ0kDXWqndlJ0iBrMuySbE3ycJIDSa7raU+ST3Tt9yW5dA41bEryzSQPJnkgyR/19Lk8yVNJ9nWPD8+6ju44jyW5vzvG3p721RiPi0b+nfuSPJ3kg2N95jIeSW5IcjjJ/pF1r0hyR5JHuueXT9j2hH9LM6jjr5M81I37zUnOnLDtCd/DGdTxkSQ/Ghn7bRO2nfd4fHGkhseS7Juw7czGY8Wqak09gHXAD4FfBE4H7gUuHuuzDbgVCPAG4LtzqONc4NJu+QzgBz11XA788yqMyWPAWSdon/t49LxH/8bSb5rmPh7ArwOXAvtH1v0VcF23fB3w0SF/SzOo4+3AS7rlj/bVsZL3cAZ1fAT4kxW8b3Mdj7H2jwEfnvd4rPSxFs/stgAHqurRqnoO+AJw1Vifq4DP1ZK7gDOTnDvLIqrqUFXd0y3/DHgQ2DjLY8zQ3MdjzFuAH1ZV3w+/Z66q7gR+Orb6KuCz3fJngd/s2XQlf0tT1VFVt1fVke7lXcD5Q/c/TR0rNPfxOCpJgN8Bbhq6/1lbi2G3EXh85PVBjg+ZlfSZmSQXAr8MfLen+Y1J7k1ya5LXzqmEAm5PcneSnT3tqzoewHYm/xGvxngAnFNVh2DpPybg7J4+qz0u72PpDLvPcu/hLFzbXU7fMOGyfjXH403AE1X1yIT21RiPY6zFsEvPuvGvjFfSZyaS/DzwJeCDVfX0WPM9LF3KvR74W+Ar86gBuKyqLgWuAD6Q5NfHy+zZZl7jcTrwTuCfeppXazxWajXH5UPAEeDzE7os9x5O61PAa4BLgEMsXUIeV2bPunn9HGMHJz6rm/d4HGctht1BYNPI6/OBxQF9ppbkNJaC7vNV9eXx9qp6uqqe6Zb3AKclOWvWdVTVYvd8GLiZpcuRUasyHp0rgHuq6omeOldlPDpPHL1U754P9/RZrb+Tq4ErgXdX94HUuBW8h1Opqieq6vmq+l/g0xP2v1rj8RLgXcAXJ/WZ93j0WYth931gc5JXd2cR24FbxvrcAryn+xbyDcBTRy9pZqX7zOEzwINV9fEJfV7Z9SPJFpbG88kZ1/GyJGccXWbpA/H9Y93mPh4jJv6PvRrjMeIW4Opu+Wrgqz19VvK3NJUkW4E/Bd5ZVc9O6LOS93DaOkY/o/2tCfuf+3h03go8VFUH+xpXYzx6rea3ISt9sPTt4g9Y+uboQ926a4BruuUAn+za7wcW5lDDr7F0in8fsK97bBur41rgAZa+1boL+NU51PGL3f7v7Y71goxHd5yXshRevzCybu7jwVK4HgL+h6Wzk/cD64FvAI90z6/o+p4H7DnR39KM6zjA0udgR/9Gdo3XMek9nHEdf9+99/exFGDnvhDj0a2/8ejfxEjfuY3HSh/eQSGpCWvxMlaSZs6wk9QEw05SEww7SU0w7CQ1wbCT1ATDTlITDDtJTfg/HsOrXkqJhdMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt \n",
    "plt.figure(figsize=(5,5))\n",
    "plt.imshow(subsequent_mask(20)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aa7394f",
   "metadata": {},
   "source": [
    "## 规范化层\n",
    "规范化层的作用：它是所有深层网络模型都需要的标准网络层，因为随着网络层数的增加，通过多层的计算后输出可能开始出现过大或过小的情况，这样可能会导致学习过程出现异常，模型可能收敛非常慢。因此都会在一定层后接规范化层进行数值的规范化，使其特征数值在合理范围内。\n",
    "\n",
    "Transformer中使用的normalization手段是layer norm，实现代码很简单，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "32f916b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    \"Construct a layernorm module (See citation for details).\"\n",
    "\n",
    "    def __init__(self, feature_size, eps=1e-6):\n",
    "        # 初始化函数有两个参数，一个是features,表示词嵌入的维度,另一个是eps它是一个足够小的数，在规范化公式的分母中出现,防止分母为0，默认是1e-6。\n",
    "        super(LayerNorm, self).__init__()\n",
    "        # 根据features的形状初始化两个参数张量a2，和b2，第一初始化为1张量，也就是里面的元素都是1，第二个初始化为0张量，也就是里面的元素都是0，这两个张量就是规范化层的参数。因为直接对上一层得到的结果做规范化公式计算，将改变结果的正常表征，因此就需要有参数作为调节因子，使其即能满足规范化要求，又能不改变针对目标的表征，最后使用nn.parameter封装，代表他们是模型的参数\n",
    "        self.a_2 = nn.Parameter(torch.ones(feature_size))\n",
    "        self.b_2 = nn.Parameter(torch.zeros(feature_size))\n",
    "        # 把eps传到类中\n",
    "        self.eps = eps\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 输入参数x代表来自上一层的输出，在函数中，首先对输入变量x求其最后一个维度的均值，并保持输出维度与输入维度一致，接着再求最后一个维度的标准差，然后就是根据规范化公式，用x减去均值除以标准差获得规范化的结果。\n",
    "        # 最后对结果乘以我们的缩放参数，即a2,*号代表同型点乘，即对应位置进行乘法操作，加上位移参b2，返回即可\n",
    "        mean = x.mean(-1, keepdim=True)\n",
    "        std = x.std(-1, keepdim=True)\n",
    "        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bc773004",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[[  0.0000,   0.3469,  -0.7491,   0.6070,  -1.5492,  -2.8974,  -3.5907,\n",
       "             0.2504,  -2.1471,   5.5847,  11.6371,  -1.9434,   0.0000,  -4.6232,\n",
       "             0.2133,  -0.0000],\n",
       "          [  0.0000,  -5.4573,  -3.7686,   1.9659,   5.1254,  -0.9123,  -1.1379,\n",
       "            -1.4928,   4.8002,   8.7096,   1.1290,   3.6947,  -2.1815,   0.8283,\n",
       "             6.0025,   4.5009],\n",
       "          [  3.0152,  -6.5201,  -3.4573,   1.8061,   0.0000,  -0.9289,  -1.1028,\n",
       "            -1.4945,   0.0000,   8.7094,   1.1325,   3.6947,  -2.1804,   0.0000,\n",
       "             6.0029,   4.5009],\n",
       "          [ -4.5933,  -5.5700,  -0.1780,  -2.9321, -10.2470,   5.1914,   0.8832,\n",
       "            -1.1694,  -5.3230,   1.8775,  -0.0000,  -6.8561,  -2.1840,   2.4458,\n",
       "            -1.7266,   3.9137],\n",
       "          [  6.2895,  -1.4905,   0.3104,  -0.1695,  -1.1165,  -2.9851,  -3.4505,\n",
       "             0.2415,  -2.1027,   5.5838,  11.6511,  -1.9435,   1.1290,  -4.6233,\n",
       "             0.2147,  -0.2102]]], grad_fn=<MulBackward0>),\n",
       " tensor([[[-0.0185,  0.0718, -0.2137,  0.1396, -0.4220, -0.7732, -0.9537,\n",
       "            0.0467, -0.5777,  1.4360,  3.0123, -0.5247, -0.0185, -1.2226,\n",
       "            0.0370, -0.0185],\n",
       "          [-0.3524, -1.7637, -1.3270,  0.1559,  0.9730, -0.5883, -0.6467,\n",
       "           -0.7385,  0.8889,  1.8998, -0.0605,  0.6030, -0.9166, -0.1382,\n",
       "            1.1998,  0.8115],\n",
       "          [ 0.5828, -1.9529, -1.1384,  0.2613, -0.2190, -0.4660, -0.5123,\n",
       "           -0.6164, -0.2190,  2.0971,  0.0822,  0.7635, -0.7988, -0.2190,\n",
       "            1.3773,  0.9779],\n",
       "          [-0.7073, -0.9424,  0.3553, -0.3075, -2.0680,  1.6475,  0.6107,\n",
       "            0.1167, -0.8829,  0.8500,  0.3981, -1.2519, -0.1275,  0.9867,\n",
       "           -0.0174,  1.3400],\n",
       "          [ 1.4046, -0.4693, -0.0356, -0.1511, -0.3793, -0.8293, -0.9415,\n",
       "           -0.0521, -0.6168,  1.2347,  2.6961, -0.5785,  0.1616, -1.2239,\n",
       "           -0.0586, -0.1609]]], grad_fn=<AddBackward0>))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# embedding_size=16\n",
    "layer_norm = LayerNorm(16)\n",
    "norm_x = layer_norm(position_enc_x)\n",
    "position_enc_x, norm_x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfcba998",
   "metadata": {},
   "source": [
    "## Attention\n",
    "Attention功能可以描述为将query和一组key-value映射到输出，其中query、key、value和输出都是向量。输出为value的加权和，其中每个value的权重通过query与相应key的计算得到。                                                                         \n",
    "我们将particular attention称之为“缩放的点积Attention”(Scaled Dot-Product Attention\")。其输入为query、key(维度是$d_k$)以及values(维度是$d_v$)。我们计算query和所有key的点积，然后对每个除以 $\\sqrt{d_k}$, 最后用softmax函数获得value的权重。         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e3242872",
   "metadata": {},
   "outputs": [],
   "source": [
    "def attention(query, key, value, mask=None, dropout=None):\n",
    "    \"Compute 'Scaled Dot Product Attention'\"\n",
    "    # query,key,value均为[batch_size,sentence_len,embedding_size]\n",
    "    # 首先取query的最后一维的大小，对应词嵌入维度\n",
    "    d_k = query.size(-1)\n",
    "    # 按照注意力公式，将query与key的转置相乘，这里面key是将最后两个维度进行转置，再除以缩放系数得到注意力得分张量scores\n",
    "    # src:scores.shape=[30,8,10,10] tgt:[30,8,9,9]\n",
    "    scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)\n",
    "\n",
    "    # 接着判断是否使用掩码张量\n",
    "    if mask is not None:\n",
    "        # 使用tensor的masked_fill方法，将掩码张量和scores张量每个位置一一比较，如果掩码张量则对应的scores张量用-1e9这个置来替换\n",
    "        scores = scores.masked_fill(mask == 0, -1e9)\n",
    "\n",
    "    # 对scores的最后一维进行softmax操作，使用F.softmax方法，这样获得最终的注意力张量\n",
    "    p_attn = F.softmax(scores, dim=-1)\n",
    "\n",
    "    # 之后判断是否使用dropout进行随机置0\n",
    "    if dropout is not None:\n",
    "        p_attn = dropout(p_attn)\n",
    "\n",
    "    # 最后，根据公式将p_attn与value张量相乘获得最终的query注意力表示，同时返回注意力张量\n",
    "    return torch.matmul(p_attn, value), p_attn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a66c648f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 5, 16])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "position_enc_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "313a8729",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 5, 16]), torch.Size([1, 5, 5]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# query的最后一维的大小为embedding_size,理论上q,k,v通过input_x线性转换后得来的.\n",
    "query, key, value = [torch.randn((1, 5, 16)) for i in range(3)]\n",
    "output_x = attention(query,key,value)\n",
    "output_x[0].shape,output_x[1].shape\n",
    "# p_attn\n",
    "# [[a11,a12,a13,a14,a15],\n",
    "#  [a21,a22,a23,a24,a25],\n",
    "#  [a31,a32,a33,a34,a35],\n",
    "#  [a41,a42,a43,a44,a45],\n",
    "#  [a51,a52,a53,a54,a55]]\n",
    "# 其中a(i,j)表示第i个单词对第j个单词的注意力程度."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed43dd4e",
   "metadata": {},
   "source": [
    "## 多头注意力机制\n",
    "\n",
    "Transformer 的论文通过增加多头注意力机制（一组注意力称为一个 attention head），进一步完善了Self-Attention。这种机制从如下两个方面增强了attention层的能力：\n",
    "\n",
    "- **它扩展了模型关注不同位置的能力**。在上面的例子中，第一个位置的输出 $z_1$ 包含了句子中其他每个位置的很小一部分信息，但$z_1$​仅仅是单个向量，所以可能仅由第1个位置的信息主导了。而当我们翻译句子：`The animal didn’t cross the street because it was too tired`时，我们不仅希望模型关注到\"it\"本身，还希望模型关注到\"The\"和\"animal\"，甚至关注到\"tired\"。这时，多头注意力机制会有帮助。\n",
    "- **多头注意力机制赋予attention层多个“子表示空间”**。下面我们会看到，多头注意力机制会有多组 $W^Q, W^K W^V$ 的权重矩阵（在 Transformer 的论文中，使用了 8 组注意力),，因此可以将 $X$ 变换到更多种子空间进行表示。接下来我们也使用8组注意力头（attention heads））。每一组注意力的权重矩阵都是随机初始化的，但经过训练之后，每一组注意力的权重 $W^Q, W^K W^V$ 可以把输入的向量映射到一个对应的\"子表示空间\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "36078435",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 12, 300])\n"
     ]
    }
   ],
   "source": [
    "class MultiheadAttention(nn.Module):\n",
    "    # n_heads：多头注意力的数量\n",
    "    # hid_dim：每个词输出的向量维度\n",
    "    def __init__(self, hid_dim, n_heads, dropout):\n",
    "        super(MultiheadAttention, self).__init__()\n",
    "        self.hid_dim = hid_dim\n",
    "        self.n_heads = n_heads\n",
    "\n",
    "        # 强制 hid_dim 必须整除 h\n",
    "        assert hid_dim % n_heads == 0\n",
    "        # 定义 W_q 矩阵\n",
    "        self.w_q = nn.Linear(hid_dim, hid_dim)\n",
    "        # 定义 W_k 矩阵\n",
    "        self.w_k = nn.Linear(hid_dim, hid_dim)\n",
    "        # 定义 W_v 矩阵\n",
    "        self.w_v = nn.Linear(hid_dim, hid_dim)\n",
    "        \n",
    "        self.fc = nn.Linear(hid_dim, hid_dim)\n",
    "        self.do = nn.Dropout(dropout)\n",
    "        # 缩放\n",
    "        self.scale = torch.sqrt(torch.FloatTensor([hid_dim // n_heads]))\n",
    "\n",
    "    def forward(self, query, key, value, mask=None):\n",
    "        # 注意 Q，K，V的在句子长度这一个维度的数值可以一样，可以不一样。\n",
    "        # K: [64,10,300], 假设batch_size 为 64，有 10 个词，每个词的 Query 向量是 300 维\n",
    "        # V: [64,10,300], 假设batch_size 为 64，有 10 个词，每个词的 Query 向量是 300 维\n",
    "        # Q: [64,12,300], 假设batch_size 为 64，有 12 个词，每个词的 Query 向量是 300 维\n",
    "        bsz = query.shape[0]\n",
    "        Q = self.w_q(query)\n",
    "        K = self.w_k(key)\n",
    "        V = self.w_v(value)\n",
    "        # 这里把 K Q V 矩阵拆分为多组注意力\n",
    "        # 最后一维就是是用 self.hid_dim // self.n_heads 来得到的，表示每组注意力的向量长度, 每个 head 的向量长度是：300/6=50\n",
    "        # 64 表示 batch size，6 表示有 6组注意力，10 表示有 10 词，50 表示每组注意力的词的向量长度\n",
    "        # K: [64,10,300] 拆分多组注意力 -> [64,10,6,50] 转置得到 -> [64,6,10,50]\n",
    "        # V: [64,10,300] 拆分多组注意力 -> [64,10,6,50] 转置得到 -> [64,6,10,50]\n",
    "        # Q: [64,12,300] 拆分多组注意力 -> [64,12,6,50] 转置得到 -> [64,6,12,50]\n",
    "        # 转置是为了把注意力的数量 6 放到前面，把 10 和 50 放到后面，方便下面计算\n",
    "        Q = Q.view(bsz, -1, self.n_heads, self.hid_dim //\n",
    "                   self.n_heads).permute(0, 2, 1, 3)\n",
    "        K = K.view(bsz, -1, self.n_heads, self.hid_dim //\n",
    "                   self.n_heads).permute(0, 2, 1, 3)\n",
    "        V = V.view(bsz, -1, self.n_heads, self.hid_dim //\n",
    "                   self.n_heads).permute(0, 2, 1, 3)\n",
    "\n",
    "        # 第 1 步：Q 乘以 K的转置，除以scale\n",
    "        # [64,6,12,50] * [64,6,50,10] = [64,6,12,10]\n",
    "        # attention：[64,6,12,10]\n",
    "        attention = torch.matmul(Q, K.permute(0, 1, 3, 2)) / self.scale\n",
    "\n",
    "        # 如果 mask 不为空，那么就把 mask 为 0 的位置的 attention 分数设置为 -1e10，这里用“0”来指示哪些位置的词向量不能被attention到，比如padding位置，当然也可以用“1”或者其他数字来指示，主要设计下面2行代码的改动。\n",
    "        if mask is not None:\n",
    "            attention = attention.masked_fill(mask == 0, -1e10)\n",
    "\n",
    "        # 第 2 步：计算上一步结果的 softmax，再经过 dropout，得到 attention。\n",
    "        # 注意，这里是对最后一维做 softmax，也就是在输入序列的维度做 softmax\n",
    "        # attention: [64,6,12,10]\n",
    "        attention = self.do(torch.softmax(attention, dim=-1))\n",
    "\n",
    "        # 第三步，attention结果与V相乘，得到多头注意力的结果\n",
    "        # [64,6,12,10] * [64,6,10,50] = [64,6,12,50]\n",
    "        # x: [64,6,12,50]\n",
    "        x = torch.matmul(attention, V)\n",
    "\n",
    "        # 因为 query 有 12 个词，所以把 12 放到前面，把 50 和 6 放到后面，方便下面拼接多组的结果\n",
    "        # x: [64,6,12,50] 转置-> [64,12,6,50]\n",
    "        x = x.permute(0, 2, 1, 3).contiguous()\n",
    "        # 这里的矩阵转换就是：把多组注意力的结果拼接起来\n",
    "        # 最终结果就是 [64,12,300]\n",
    "        # x: [64,12,6,50] -> [64,12,300]\n",
    "        x = x.view(bsz, -1, self.n_heads * (self.hid_dim // self.n_heads))\n",
    "        x = self.fc(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# batch_size 为 64，有 12 个词，每个词的 Query 向量是 300 维\n",
    "query = torch.rand(64, 12, 300)\n",
    "# batch_size 为 64，有 12 个词，每个词的 Key 向量是 300 维\n",
    "key = torch.rand(64, 12, 300)\n",
    "# batch_size 为 64，有 12 个词，每个词的 Value 向量是 300 维\n",
    "value = torch.rand(64, 12, 300)\n",
    "attention_fn = MultiheadAttention(hid_dim=300, n_heads=6, dropout=0.1)\n",
    "output = attention_fn(query, key, value)\n",
    "## output: torch.Size([64, 12, 300])\n",
    "print(output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e121fc2e",
   "metadata": {},
   "source": [
    "### 简化代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "558fad5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一个clones函数，来更方便的将某个结构复制若干份\n",
    "def clones(module, N):\n",
    "    \"Produce N identical layers.\"\n",
    "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7df55e18",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadedAttention(nn.Module):\n",
    "    def __init__(self, h, d_model, dropout=0.1):\n",
    "        # 在类的初始化时，会传入三个参数，h代表头数，d_model代表词嵌入的维度，dropout代表进行dropout操作时置0比率，默认是0.1\n",
    "        super(MultiHeadedAttention, self).__init__()\n",
    "        # 在函数中，首先使用了一个测试中常用的assert语句，判断h是否能被d_model整除，这是因为我们之后要给每个头分配等量的词特征，也就是embedding_dim/head个\n",
    "        assert d_model % h == 0\n",
    "        # 得到每个头获得的分割词向量维度d_k\n",
    "        self.d_k = d_model // h\n",
    "        # 传入头数h\n",
    "        self.h = h\n",
    "\n",
    "        # 创建linear层，通过nn的Linear实例化，它的内部变换矩阵是embedding_dim x embedding_dim，然后使用，为什么是四个呢，这是因为在多头注意力中，Q,K,V各需要一个，最后拼接的矩阵还需要一个，因此一共是四个\n",
    "        self.linears = clones(nn.Linear(d_model, d_model), 4)\n",
    "        # self.attn为None，它代表最后得到的注意力张量，现在还没有结果所以为None\n",
    "        self.attn = None\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "\n",
    "    def forward(self, query, key, value, mask=None):\n",
    "        # 前向逻辑函数，它输入参数有四个，前三个就是注意力机制需要的Q,K,V，最后一个是注意力机制中可能需要的mask掩码张量，默认是None\n",
    "        if mask is not None:\n",
    "            # Same mask applied to all h heads.\n",
    "            # 使用unsqueeze扩展维度，代表多头中的第n头\n",
    "            mask = mask.unsqueeze(1)\n",
    "        # 接着，我们获得一个batch_size的变量，他是query尺寸的第1个数字，代表有多少条样本\n",
    "        nbatches = query.size(0)\n",
    "\n",
    "        # 1) Do all the linear projections in batch from d_model => h x d_k\n",
    "        # 首先利用zip将输入QKV与三个线性层组到一起，然后利用for循环，将输入QKV分别传到线性层中，做完线性变换后，开始为每个头分割输入，这里使用view方法对线性变换的结构进行维度重塑，多加了一个维度h代表头，这样就意味着每个头可以获得一部分词特征组成的句子，其中的-1代表自适应维度，计算机会根据这种变换自动计算这里的值，然后对第二维和第三维进行转置操作，为了让代表句子长度维度和词向量维度能够相邻，这样注意力机制才能找到词义与句子位置的关系，从attention函数中可以看到，利用的是原始输入的倒数第一和第二维，这样我们就得到了每个头的输入\n",
    "        # src:query.shape=[30,10,512] tgt:query.shape=[30,9,512]\n",
    "        query, key, value = \\\n",
    "            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n",
    "             for l, x in zip(self.linears, (query, key, value))]\n",
    "\n",
    "        # 2) Apply attention on all the projected vectors in batch.\n",
    "        # 得到每个头的输入后，接下来就是将他们传入到attention中，这里直接调用我们之前实现的attention函数，同时也将mask和dropout传入其中\n",
    "        # src:query.shape=[30,8,10,64] tgt:query.shape=[30,8,9,64]\n",
    "        # 8*64 = 512\n",
    "        x, self.attn = attention(\n",
    "            query, key, value, mask=mask, dropout=self.dropout)\n",
    "\n",
    "        # 3) \"Concat\" using a view and apply a final linear.\n",
    "        # 通过多头注意力计算后，我们就得到了每个头计算结果组成的4维张量，我们需要将其转换为输入的形状以方便后续的计算，因此这里开始进行第一步处理环节的逆操作，先对第二和第三维进行转置，然后使用contiguous方法。这个方法的作用就是能够让转置后的张量应用view方法，否则将无法直接使用，所以，下一步就是使用view重塑形状，变成和输入形状相同。\n",
    "        x = x.transpose(1, 2).contiguous() \\\n",
    "             .view(nbatches, -1, self.h * self.d_k)\n",
    "        # 最后使用线性层列表中的最后一个线性变换得到最终的多头注意力结构的输出\n",
    "        # src:return.shape=[30,10,512] tgt:return.shape=[30,9,512]\n",
    "        return self.linears[-1](x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd344ad",
   "metadata": {},
   "source": [
    " ## 前馈全连接层\n",
    "除了attention子层之外，我们的编码器和解码器中的每个层都包含一个全连接的前馈网络，该网络在每个层的位置相同（都在每个encoder-layer或者decoder-layer的最后）。该前馈网络包括两个线性变换，并在两个线性变换中间有一个ReLU激活函数。\n",
    "\n",
    "$$\\mathrm{FFN}(x)=\\max(0, xW_1 + b_1) W_2 + b_2$$                                                                        \n",
    "\n",
    "尽管两层都是线性变换，但它们在层与层之间使用不同的参数。另一种描述方式是两个内核大小为1的卷积。 输入和输出的维度都是 $d_{\\text{model}}=512$, 内层维度是$d_{ff}=2048$。（也就是第一层输入512维,输出2048维；第二层输入2048维，输出512维）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cbdf4d0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionwiseFeedForward(nn.Module):\n",
    "    def __init__(self, d_model, d_ff, dropout=0.1):\n",
    "        #初始化函数有三个输入参数分别是d_model，d_ff，和dropout=0.1，第一个是线性层的输入维度也是第二个线性层的输出维度，因为我们希望输入通过前馈全连接层后输入和输出的维度不变，第二个参数d_ff就是第二个线性层的输入维度和第一个线性层的输出，最后一个是dropout置0比率。\n",
    "        super(PositionwiseFeedForward, self).__init__()\n",
    "        self.w_1 = nn.Linear(d_model, d_ff)\n",
    "        self.w_2 = nn.Linear(d_ff, d_model)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        #输入参数为x，代表来自上一层的输出，首先经过第一个线性层，然后使用F中的relu函数进行激活，之后再使用dropout进行随机置0，最后通过第二个线性层w2，返回最终结果\n",
    "        return self.w_2(self.dropout(F.relu(self.w_1(x))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a4cf4d",
   "metadata": {},
   "source": [
    "## Encoder\n",
    "编码器由N = 6个完全相同的层组成。\n",
    "编码器的每层encoder包含Self Attention 子层和FFN子层，每个子层都使用了残差连接[(cite)](https://arxiv.org/abs/1512.03385)，和层标准化（layer-normalization） [(cite)](https://arxiv.org/abs/1607.06450)。\n",
    "\n",
    "我们称呼子层为：$\\mathrm{Sublayer}(x)$，每个子层的最终输出是$\\mathrm{LayerNorm}(x + \\mathrm{Sublayer}(x))$。 dropout [(cite)](http://jmlr.org/papers/v15/srivastava14a.html)被加在Sublayer上。\n",
    "\n",
    "为了便于进行残差连接，模型中的所有子层以及embedding层产生的输出的维度都为 $d_{\\text{model}}=512$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "456324cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一个clones函数，来更方便的将某个结构复制若干份\n",
    "def clones(module, N):\n",
    "    \"Produce N identical layers.\"\n",
    "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])\n",
    "\n",
    "\n",
    "class Encoder(nn.Module):\n",
    "    \"\"\"\n",
    "    Encoder\n",
    "    The encoder is composed of a stack of N=6 identical layers.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, layer, N):\n",
    "        super(Encoder, self).__init__()\n",
    "        # 调用时会将编码器层传进来，我们简单克隆N分，叠加在一起，组成完整的Encoder\n",
    "        self.layers = clones(layer, N)\n",
    "        self.norm = LayerNorm(layer.size)\n",
    "\n",
    "    def forward(self, x, mask):\n",
    "        \"Pass the input (and mask) through each layer in turn.\"\n",
    "        for layer in self.layers:\n",
    "            x = layer(x, mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4978cf74",
   "metadata": {},
   "source": [
    "第一个子层包括一个**多头自注意力层**和**规范化层**以及一个**残差连接**\n",
    "\n",
    "第二个子层包括一个**前馈全连接层**和**规范化层**以及一个**残差连接**\n",
    "\n",
    "可以看到，两个子层的结构其实是一致的，只是中间核心层的实现不同.\n",
    "\n",
    "![image-20211014164210704](https://gitee.com/shenhao-stu/picgo/raw/master/Others/image-20211014164210704.png)\n",
    "\n",
    "下面的**SublayerConnection**类用来处理单个Sublayer的输出，该输出将继续被输入下一个Sublayer："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "bdcf110d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SublayerConnection(nn.Module):\n",
    "    \"\"\" \n",
    "    实现子层连接结构的类\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, size, dropout):\n",
    "        super(SublayerConnection, self).__init__()\n",
    "        self.norm = LayerNorm(size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, sublayer):\n",
    "\n",
    "        # 原paper的方案\n",
    "        #sublayer_out = sublayer(x)\n",
    "        #x_norm = self.norm(x + self.dropout(sublayer_out))\n",
    "\n",
    "        # 稍加调整的版本\n",
    "        sublayer_out = sublayer(x)\n",
    "        sublayer_out = self.dropout(sublayer_out)\n",
    "        x_norm = x + self.norm(sublayer_out)\n",
    "        return x_norm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea26bf10",
   "metadata": {},
   "source": [
    "注：上面的实现中，我对残差的链接方案进行了小小的调整，和原论文有所不同。把x从norm中拿出来，保证永远有一条“高速公路”，这样理论上会收敛的快一些，但我无法确保这样做一定是对的，请一定注意。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04340eed",
   "metadata": {},
   "source": [
    "定义好了SubLayerConnection，我们就可以实现EncoderLayer的结构了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "aef53a0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayer(nn.Module):\n",
    "    \"EncoderLayer is made up of two sublayer: self-attn and feed forward\"\n",
    "\n",
    "    def __init__(self, size, self_attn, feed_forward, dropout):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "        self.self_attn = self_attn\n",
    "        self.feed_forward = feed_forward\n",
    "        self.sublayer = clones(SublayerConnection(size, dropout), 2)\n",
    "        self.size = size   # embedding's dimention of model, 默认512\n",
    "\n",
    "    def forward(self, x, mask):\n",
    "        # attention sub layer,let self.self_attn use one arg by using lambda\n",
    "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))\n",
    "        # feed forward sub layer\n",
    "        z = self.sublayer[1](x, self.feed_forward)\n",
    "        return z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7607b20a",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "解码器也是由N = 6 个完全相同的decoder层组成。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71378e44",
   "metadata": {},
   "source": [
    "### 1. 解码器整体结构\n",
    "解码器的作用：根据编码器的结果以及上一次预测的结果，输出序列的下一个结果。\n",
    "\n",
    "整体结构上，解码器也是由N个相同层堆叠而成。构造代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "848e558b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用类Decoder来实现解码器\n",
    "class Decoder(nn.Module):\n",
    "    \"Generic N layer decoder with masking.\"\n",
    "\n",
    "    def __init__(self, layer, N):\n",
    "        # 初始化函数的参数有两个，第一个就是解码器层layer，第二个是解码器层的个数N\n",
    "        super(Decoder, self).__init__()\n",
    "        # 首先使用clones方法克隆了N个layer，然后实例化一个规范化层，因为数据走过了所有的解码器层后最后要做规范化处理。\n",
    "        self.layers = clones(layer, N)\n",
    "        self.norm = LayerNorm(layer.size)\n",
    "\n",
    "    def forward(self, x, memory, src_mask, tgt_mask):\n",
    "        # forward函数中的参数有4个，x代表目标数据的嵌入表示，memory是编码器层的输出，source_mask，target_mask代表源数据和目标数据的掩码张量，然后就是对每个层进行循环，当然这个循环就是变量x通过每一个层的处理，得出最后的结果，再进行一次规范化返回即可。\n",
    "        for layer in self.layers:\n",
    "            x = layer(x, memory, src_mask, tgt_mask)\n",
    "        return self.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d3bd249",
   "metadata": {},
   "source": [
    "### 2. 解码器层\n",
    "每个解码器层由三个子层连接结构组成\n",
    "- 第一个子层连接结构包括一个**多头自注意力子层**和规范化层以及一个残差连接\n",
    "- 第二个子层连接结构包括一个**多头注意力子层**和规范化层以及一个残差连接\n",
    "- 第三个子层连接结构包括一个**前馈全连接子层**和规范化层以及一个残差连接。\n",
    "![image-20211014165218933](https://gitee.com/shenhao-stu/picgo/raw/master/Others/image-20211014165218933.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f57788d",
   "metadata": {},
   "source": [
    "有一个细节需要注意，第一个子层的多头注意力和编码器中完全一致，第二个子层，它的**多头注意力模块**中，**query来自上一个子层，key 和 value 来自编码器的输出**。可以这样理解，就是第二层负责，利用解码器已经预测出的信息作为query，去编码器提取的各种特征中，查找相关信息并融合到当前特征中，来完成预测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "232c5233",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用DecoderLayer的类实现解码器层\n",
    "class DecoderLayer(nn.Module):\n",
    "    \"Decoder is made of self-attn, src-attn, and feed forward (defined below)\"\n",
    "\n",
    "    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):\n",
    "        # 初始化函数的参数有5个，分别是size，代表词嵌入的维度大小，同时也代表解码器的尺寸，第二个是self_attn，多头自注意力对象，也就是说这个注意力机制需要Q=K=V，第三个是src_attn,多头注意力对象，这里Q!=K=V，第四个是前馈全连接层对象，最后就是dropout置0比率\n",
    "        super(DecoderLayer, self).__init__()\n",
    "        self.size = size\n",
    "        self.self_attn = self_attn\n",
    "        self.src_attn = src_attn\n",
    "        self.feed_forward = feed_forward\n",
    "        # 按照结构图使用clones函数克隆三个子层连接对象\n",
    "        self.sublayer = clones(SublayerConnection(size, dropout), 3)\n",
    "\n",
    "    def forward(self, x, memory, src_mask, tgt_mask):\n",
    "        # forward函数中的参数有4个，分别是来自上一层的输入x，来自编码器层的语义存储变量memory，以及源数据掩码张量和目标数据掩码张量，将memory表示成m之后方便使用。\n",
    "        m = memory\n",
    "        # 将x传入第一个子层结构，第一个子层结构的输入分别是x和self-attn函数，因为是自注意力机制，所以Q,K,V都是x，最后一个参数时目标数据掩码张量，这时要对目标数据进行遮掩，因为此时模型可能还没有生成任何目标数据。\n",
    "        # 比如在解码器准备生成第一个字符或词汇时，我们其实已经传入了第一个字符以便计算损失，但是我们不希望在生成第一个字符时模型能利用这个信息，因此我们会将其遮掩，同样生成第二个字符或词汇时，模型只能使用第一个字符或词汇信息，第二个字符以及之后的信息都不允许被模型使用。\n",
    "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))\n",
    "        # 接着进入第二个子层，这个子层中常规的注意力机制，q是输入x;k,v是编码层输出memory，同样也传入source_mask，但是进行源数据遮掩的原因并非是抑制信息泄露，而是遮蔽掉对结果没有意义的padding。\n",
    "        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))\n",
    "\n",
    "        # 最后一个子层就是前馈全连接子层，经过它的处理后就可以返回结果，这就是我们的解码器结构\n",
    "        return self.sublayer[2](x, self.feed_forward)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a1f8ad",
   "metadata": {},
   "source": [
    "## 模型输出\n",
    "\n",
    "输出部分就很简单了，每个时间步都过一个 线性层 + softmax层\n",
    "\n",
    "线性层的作用：通过对上一步的线性变化得到指定维度的输出，也就是转换维度的作用。转换后的维度对应着输出类别的个数，如果是翻译任务，那就对应的是文字字典的大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c473170b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将线性层和softmax计算层一起实现，因为二者的共同目标是生成最后的结构\n",
    "# 因此把类的名字叫做Generator，生成器类\n",
    "class Generator(nn.Module):\n",
    "    \"Define standard linear + softmax generation step.\"\n",
    "\n",
    "    def __init__(self, d_model, vocab):\n",
    "        # 初始化函数的输入参数有两个，d_model代表词嵌入维度，vocab.size代表词表大小\n",
    "        super(Generator, self).__init__()\n",
    "        # 首先就是使用nn中的预定义线性层进行实例化，得到一个对象self.proj等待使用\n",
    "        # 这个线性层的参数有两个，就是初始化函数传进来的两个参数：d_model，vocab_size\n",
    "        self.proj = nn.Linear(d_model, vocab)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 前向逻辑函数中输入是上一层的输出张量x,在函数中，首先使用上一步得到的self.proj对x进行线性变化,然后使用F中已经实现的log_softmax进行softmax处理。\n",
    "        return F.log_softmax(self.proj(x), dim=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c04d907",
   "metadata": {},
   "source": [
    "## 模型构建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "46e40095",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model Architecture\n",
    "# 使用EncoderDecoder类来实现编码器-解码器结构\n",
    "class EncoderDecoder(nn.Module):\n",
    "    \"\"\"\n",
    "    A standard Encoder-Decoder architecture. \n",
    "    Base for this and many other models.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):\n",
    "        # 初始化函数中有5个参数，分别是编码器对象，解码器对象,源数据嵌入函数，目标数据嵌入函数，以及输出部分的类别生成器对象.\n",
    "        super(EncoderDecoder, self).__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        # input embedding module(input embedding + positional encode)\n",
    "        self.src_embed = src_embed\n",
    "        self.tgt_embed = tgt_embed    # ouput embedding module\n",
    "        self.generator = generator    # output generation module\n",
    "\n",
    "    def forward(self, src, tgt, src_mask, tgt_mask):\n",
    "        \"Take in and process masked src and target sequences.\"\n",
    "        # 在forward函数中，有四个参数，source代表源数据，target代表目标数据,source_mask和target_mask代表对应的掩码张量,在函数中，将source source_mask传入编码函数，得到结果后与source_mask target 和target_mask一同传给解码函数\n",
    "        memory = self.encode(src, src_mask)\n",
    "        res = self.decode(memory, src_mask, tgt, tgt_mask)\n",
    "        return res\n",
    "\n",
    "    def encode(self, src, src_mask):\n",
    "        # 编码函数，以source和source_mask为参数,使用src_embed对source做处理，然后和source_mask一起传给self.encoder\n",
    "        src_embedds = self.src_embed(src)  # src_embedds.shape=[30,10,512]\n",
    "        return self.encoder(src_embedds, src_mask)  # src_mask.shape=[30,1,10]\n",
    "\n",
    "    def decode(self, memory, src_mask, tgt, tgt_mask):\n",
    "        # 解码函数，以memory即编码器的输出，source_mask target target_mask为参数,使用tgt_embed对target做处理，然后和source_mask,target_mask,memory一起传给self.decoder\n",
    "        target_embedds = self.tgt_embed(tgt)\n",
    "        return self.decoder(target_embedds, memory, src_mask, tgt_mask)\n",
    "\n",
    "\n",
    "# Full Model\n",
    "def make_model(src_vocab, tgt_vocab, N=6, d_model=512, d_ff=2048, h=8, dropout=0.1):\n",
    "    \"\"\"\n",
    "    构建模型\n",
    "    params:\n",
    "        src_vocab:\n",
    "        tgt_vocab:\n",
    "        N: 编码器和解码器堆叠基础模块的个数\n",
    "        d_model: 模型中embedding的size，默认512\n",
    "        d_ff: FeedForward Layer层中embedding的size，默认2048\n",
    "        h: MultiHeadAttention中多头的个数，必须被d_model整除\n",
    "        dropout:\n",
    "    \"\"\"\n",
    "    c = copy.deepcopy\n",
    "    attn = MultiHeadedAttention(h, d_model, dropout)\n",
    "    ff = PositionwiseFeedForward(d_model, d_ff, dropout)\n",
    "    position = PositionalEncoding(d_model, dropout)\n",
    "    model = EncoderDecoder(\n",
    "        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),\n",
    "        Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), N),\n",
    "        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),\n",
    "        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),\n",
    "        Generator(d_model, tgt_vocab))\n",
    "\n",
    "    # This was important from their code.\n",
    "    # Initialize parameters with Glorot / fan_avg.\n",
    "    for p in model.parameters():\n",
    "        if p.dim() > 1:\n",
    "            nn.init.xavier_uniform_(p)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1ccb5c9",
   "metadata": {},
   "source": [
    "## 实战案例\n",
    "\n",
    "下面我们用一个人造的玩具级的小任务，来实战体验下Transformer的训练，加深我们的理解，并且验证我们上面所述代码是否work。\n",
    "\n",
    "任务描述：针对数字序列进行学习，学习的最终目标是使模型学会输出与输入的序列删除第一个字符之后的相同的序列，如输入[1,2,3,4,5]，我们尝试让模型学会输出[2,3,4,5]。\n",
    "\n",
    "显然这对模型来说并不难，应该简单的若干次迭代就能学会。\n",
    "\n",
    "代码实现的基本的步骤是：\n",
    "\n",
    "第一步：构建并生成人工数据集\n",
    "\n",
    "第二步：构建Transformer模型及相关准备工作\n",
    "\n",
    "第三步：运行模型进行训练和评估\n",
    "\n",
    "第四步：使用模型进行贪婪解码\n",
    "\n",
    "训练的大致流程如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5439adc",
   "metadata": {},
   "source": [
    "### 批处理和掩码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f149775b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Batch:\n",
    "    \"Object for holding a batch of data with mask during training.\"\n",
    "\n",
    "    def __init__(self, src, trg=None, pad=0):\n",
    "        # src,trg [30,10]\n",
    "        self.src = src\n",
    "        # src_mask.shape=[30,1,10]\n",
    "        self.src_mask = (src != pad).unsqueeze(-2)\n",
    "        if trg is not None:\n",
    "            self.trg = trg[:, :-1]  # [30,9]\n",
    "            self.trg_y = trg[:, 1:]  # [30,9]\n",
    "            self.trg_mask = \\\n",
    "                self.make_std_mask(self.trg, pad)\n",
    "            self.ntokens = (self.trg_y != pad).data.sum()\n",
    "\n",
    "    @staticmethod\n",
    "    def make_std_mask(tgt, pad):\n",
    "        # tgt.shape [30,9]\n",
    "        \"Create a mask to hide padding and future words.\"\n",
    "        tgt_mask = (tgt != pad).unsqueeze(-2)  # [30,1,9]\n",
    "        tgt_mask = tgt_mask & subsequent_mask(\n",
    "            tgt.size(-1)).type_as(tgt_mask.data)  # subsequent_mask.shape [1,9,9]\n",
    "        # tgt_mask=[30,9,9]\n",
    "        return tgt_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "994c0c7b",
   "metadata": {},
   "source": [
    "### Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "7ef22bfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_epoch(data_iter, model, loss_compute):\n",
    "    \"Standard Training and Logging Function\"\n",
    "    start = time.time()\n",
    "    total_tokens = 0\n",
    "    total_loss = 0\n",
    "    tokens = 0\n",
    "    # batch.src.shape=[30,10] batch.trg.shape=[30,9]\n",
    "    for i, batch in enumerate(data_iter):\n",
    "        out = model.forward(batch.src, batch.trg,\n",
    "                            batch.src_mask, batch.trg_mask)\n",
    "        loss = loss_compute(out, batch.trg_y, batch.ntokens)\n",
    "        total_loss += loss\n",
    "        total_tokens += batch.ntokens\n",
    "        tokens += batch.ntokens\n",
    "        if i % 50 == 1:\n",
    "            elapsed = time.time() - start\n",
    "            print(\"Epoch Step: %d Loss: %f Tokens per Sec: %f\" %\n",
    "                  (i, loss / batch.ntokens, tokens / elapsed))\n",
    "            start = time.time()\n",
    "            tokens = 0\n",
    "    return total_loss / total_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c76e5d",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "我们使用Adam优化器[(cite)](https://arxiv.org/abs/1412.6980)，其中 $\\beta_1=0.9$, $\\beta_2=0.98$并且$\\epsilon=10^{-9}$。我们根据以下公式在训练过程中改变学习率：                                         \n",
    "$$\n",
    "lrate = d_{\\text{model}}^{-0.5} \\cdot                                                                                                                                                                                                                                                                                                \n",
    "  \\min({step\\_num}^{-0.5},                                                                                                                                                                                                                                                                                                  \n",
    "    {step\\_num} \\cdot {warmup\\_steps}^{-1.5})                                                                                                                                                                                                                                                                               \n",
    "$$\n",
    "这对应于在第一次$warmup\\_steps$步中线性地增加学习速率，并且随后将其与步数的平方根成比例地减小。我们使用$warmup\\_steps=4000$。                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "089b3ecb",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NoamOpt:\n",
    "    \"Optim wrapper that implements rate.\"\n",
    "\n",
    "    def __init__(self, model_size, factor, warmup, optimizer):\n",
    "        self.optimizer = optimizer\n",
    "        self._step = 0\n",
    "        self.warmup = warmup\n",
    "        self.factor = factor\n",
    "        self.model_size = model_size\n",
    "        self._rate = 0\n",
    "\n",
    "    def step(self):\n",
    "        \"Update parameters and rate\"\n",
    "        self._step += 1\n",
    "        rate = self.rate()\n",
    "        for p in self.optimizer.param_groups:\n",
    "            p['lr'] = rate\n",
    "        self._rate = rate\n",
    "        self.optimizer.step()\n",
    "\n",
    "    def rate(self, step=None):\n",
    "        \"Implement `lrate` above\"\n",
    "        if step is None:\n",
    "            step = self._step\n",
    "        return self.factor * \\\n",
    "            (self.model_size ** (-0.5) *\n",
    "             min(step ** (-0.5), step * self.warmup ** (-1.5)))\n",
    "\n",
    "\n",
    "def get_std_opt(model):\n",
    "    return NoamOpt(model.src_embed[0].d_model, 2, 4000,\n",
    "                   torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba9b8f2d",
   "metadata": {},
   "source": [
    "> 以下是此模型针对不同模型大小和优化超参数的曲线示例。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6baef6c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x272213e0be0>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD4CAYAAAD2FnFTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABN40lEQVR4nO3dd3hUVfrA8e9Jh/RCeiUJJRBKCL2DIAiKgCCK2FDEsrqWXV13dV1XV1Zd28rqT10F1BVRQVEQEJiAhF5CgEAgIT0hlXRS5/z+mCEGSBmSSSblfJ4nT2buvefed4Yw75x7z32PkFKiKIqiKFczM3UAiqIoSsekEoSiKIrSIJUgFEVRlAapBKEoiqI0SCUIRVEUpUEWpg7AGNzc3GRgYKCpw1AURelUjhw5kiel7NXY+i6RIAIDAzl8+LCpw1AURelUhBApTa1Xp5gURVGUBqkEoSiKojRIJQhFURSlQQZdgxBCzADeBcyBT6SUK65aL/TrbwLKgXullEebaiuEWAC8BPQHRkgpD9fb35+ApUAt8LiUcmsrXqOiKJ1QdXU16enpVFRUmDqUTs/GxgZfX18sLS2vq12zCUIIYQ6sBKYB6cAhIcRGKWVcvc1mAqH6n5HAB8DIZtqeBOYB/3fV8cKARcAAwBvYLoToI6Wsva5XpihKp5aeno69vT2BgYHovoMqLSGlJD8/n/T0dIKCgq6rrSGnmEYACVLK81LKKmAtMOeqbeYAa6TOfsBJCOHVVFsp5WkpZXwDx5sDrJVSVkopk4AE/X4URelGKioqcHV1VcmhlYQQuLq6tqgnZkiC8AHS6j1P1y8zZBtD2rbkeAghlgkhDgshDufm5jazS0VROiOVHIyjpe+jIQmioT1fXSO8sW0MaduS4yGl/EhKGSmljOzVq9H7PJSrlFaV8s3Zb6jR1pg6FEVROjhDEkQ64FfvuS+QaeA2hrRtyfGUFlodt5qX973Mf0/819ShKEqnEBgYSHh4OEOGDCEyMhKAb775hgEDBmBmZnbFTbq//PILw4YNIzw8nGHDhrFz584m9/3mm28ihCAvL69u2WuvvUZISAh9+/Zl69bfxuccOXKE8PBwQkJCePzxx7k8l09lZSW33347ISEhjBw5kuTkZKO9dkMSxCEgVAgRJISwQncBeeNV22wE7hY6o4AiKWWWgW2vthFYJISwFkIEobvwffA6XpPShANZBwD4+MTHpBanmjgaRekcNBoNMTExdclg4MCBrF+/ngkTJlyxnZubGz/++CMnTpxg9erVLFmypNF9pqWl8csvv+Dv71+3LC4ujrVr13Lq1Cm2bNnCI488Qm2tbnzOww8/zEcffcS5c+c4d+4cW7ZsAeC///0vzs7OJCQk8OSTT/Lss88a7XU3myCklDXAY8BW4DSwTkp5SgixXAixXL/ZZuA8ugvKHwOPNNUWQAgxVwiRDowGNgkhturbnALWAXHAFuBRNYLJOPIv5ROTE8P80PlYmlnyt31/q/sWoiiK4fr370/fvn2vWT506FC8vb0BGDBgABUVFVRWVja4jyeffJLXX3/9iusDP/zwA4sWLcLa2pqgoCBCQkI4ePAgWVlZFBcXM3r0aIQQ3H333Xz//fd1be655x4AbrvtNnbs2GG0/9cG3QchpdyMLgnUX/ZhvccSeNTQtvrlG4ANjbR5FXjVkNgUw+1O341Ecnvf2xngNoCX973MhoQNzAudZ+rQFKVJf/vxFHGZxUbdZ5i3A3+9eUCz2wkhmD59OkIIHnroIZYtW2bQ/r/77juGDh2KtbU1AA888ADLly8nMjKSjRs34uPjw+DBg69ok5GRwahRo+qe+/r6kpGRgaWlJb6+vtcsv9zGz093Vt7CwgJHR0fy8/Nxc3MzKM6mdIlifYphdqbtxMvWi34u/ejr0pdN5zfx5uE3Ge8znl491YV+RWlIdHQ03t7e5OTkMG3aNPr163fNqaWrnTp1imeffZZt27bVLfvkk08AKC8v59VXX71i3WUNffMXQjS6vKk2xqASRDdxqeYS+zP3Mzd0LkIIBIKXRr/E/I3zeXn/y7w3+T01pFDpsAz5pt9WLp8ycnd3Z+7cuRw8eLDJBJGens7cuXNZs2YNwcHB16xPTEwkKSmprveQnp5OREQEBw8exNfXl7S0tCv25e3tja+vL+np6dcsB+ra+Pr6UlNTQ1FRES4uLkZ57aoWUzexL3MfFbUVTPabXLcs0DGQxyMeJyotiu8TvjdZbIrSUZWVlVFSUlL3eNu2bQwcOLDR7QsLC5k1axavvfYaY8eObXCb8PBwcnJySE5OJjk5GV9fX44ePYqnpye33HILa9eupbKykqSkJM6dO8eIESPw8vLC3t6e/fv3I6VkzZo1zJmju1/5lltuYfXq1QB8++23TJkyxWhf9lSC6Cai0qKwt7Qn0jPyiuVLwpYwwnMEKw6uIK0kreHGitJNZWdnM27cOAYPHsyIESOYNWsWM2bMYMOGDfj6+rJv3z5mzZrFjTfeCMD7779PQkICf//73xkyZAhDhgwhJycH0F2DaG7emgEDBrBw4ULCwsKYMWMGK1euxNzcHIAPPviABx54gJCQEIKDg5k5cyYAS5cuJT8/n5CQEN566y1WrFjR1CGui+gKo1giIyOlmjCocbXaWqZ8M4WRXiN5fcLr16zPKs1i/sb5hDiH8NmNn2FuZm6CKBXlSqdPn6Z///6mDqPLaOj9FEIckVJGNtJE9SC6g9i8WAoqCq44vVSfl50Xz496nmM5x/js1GftHJ2iKB2VShDdgCZVg4WZBeN8xjW6zaygWdwYeCMrj60kNje2HaNTFKWjUgmiG9CkaRjuMRx7K/tGtxFC8MKoF/Cw9eAPu/5AUWVRO0aoKEpHpBJEF3e+6DzJxclM9m/49FJ9jtaOvDHhDXIu5fCX6L+ou6wVpZtTCaKLi0qLAmj0+sPVwnuF8/Swp4lKi+LzuM/bLC5FUTo+lSC6OE2qhv4u/fG09TS4zeL+i5nqP5W3j7zN8dzjbRidoigdmUoQXVjepTyO5x43uPdwmRCCl8e+jIetB09pniK3XE3IpHRfbVHuOyYmhlGjRtXt8+DB3wpWd6Ry30gpO/3PsGHDpHKt785+JweuGihP559uUfsz+Wfk8C+Gy8WbFsvKmkojR6coTYuLizN1CFJKKQMCAmRubu4Vy+Li4uSZM2fkxIkT5aFDh+qWHz16VGZkZEgppTxx4oT09vZucJ/Tpk2TmzdvllJKuWnTJjlx4kQppZSnTp2SgwYNkhUVFfL8+fOyd+/esqamRkop5fDhw+XevXulVquVM2bMqGu/cuVK+dBDD0kppfzqq6/kwoULGzxmQ+8ncFg28dmqehBdmCZVg7etN32dry1LbIi+Ln15ddyrHM89zqsHXlUXrRVFr7XlvoUQFBfrqtMWFRXVtemU5b6Vzqe8upx9WfuYHzq/VXVZpgVMY9mgZXwU+xF9nftyZ/87jRilohjo5+fgwgnj7tMzHGY2X5aiLcp9v/POO9x4440888wzaLVa9u7dC6hy30o72Z+1n8raSoOGtzbn0SGPcrbgLK8fep0gxyBGe482QoSK0jkYu9w36Ooqvf3228yfP59169axdOlStm/frsp9K+1Dk6bB3tKeYR7DWr0vM2HGa+NfY8nPS3gq6ilWz1xNH+c+RohSUQxkwDf9tmLsct8Aq1ev5t133wVgwYIFPPDAAwCq3LfS9mq1texO380433FYmlkaZZ92VnZ8cMMH9LToySPbHyG7LNso+1WUjqwtyn2DLuns2rULgJ07dxIaGgrQ4cp9m3wEkjF+1CimKx25cEQOXDVQ/nz+Z6Pv+0z+GTnyy5Fy3g/zZEllidH3ryiXdYRRTImJiXLQoEFy0KBBMiwsTL7yyitSSinXr18vfXx8pJWVlXR3d5fTp0+XUkr597//Xfbs2VMOHjy47ic7O1tKKeXSpUvrRjz9+uuvMiIiQg4aNEiOGDFCHj58uO6Yr7zyiuzdu7fs06dP3UglKaU8dOiQHDBggOzdu7d89NFHpVarlVJKeenSJXnbbbfJ4OBgOXz4cJmYmNjga2nJKCZV7rsL+tfhf/HF6S/49fZfsbOyM/r+92bs5dEdjzLcczgrp67E0tw4vRRFqU+V+zYuVe5bQUqJJk3DCM8RbZIcAMb4jOHF0S+yL2sfz+95nlptbZscR1EU01IXqbuYpOIkUopTuKv/XW16nLmhcymqLOJfR/6FraUtfx39VzWntaJ0MSpBdDGaVA0Ak/wmtfmx7h14LyXVJXwU+xE9LXvyh8g/qCShKF2IShBdjCZNQ5hr2HUV52uNx4Y8Rll1GZ/HfY69pT0PD3m4XY6rKErbUwmiC8m7lEdsbmy7fkgLIfjj8D9SVl3Gf47/B0tzSx4If6Ddjq8oSttRCaIL2ZW2C4lkit+Udj2umTDjpdEvUa2t5t2j71KtrebhwaonoSidnRrF1IVo0nTF+Uxxl7O5mTmvjn2VW4Jv4T8x/+H9Y++r4n5Kp5eWlsbkyZPp378/AwYMqLv7+aWXXsLHx4chQ4YwZMgQNm/eXNcmNjaW0aNHM2DAAMLDw6moqGh0/2+++SZCCPLy8uqWqXLf6kY5oyurKpPDPh8mXzvwmknjqNXWyhejX5QDVw2U7xx5p+5mHkW5Xh3hRrnMzEx55MgRKaWUxcXFMjQ0VJ46dUr+9a9/lW+88cY121dXV8vw8HAZExMjpZQyLy+vrlz31VJTU+X06dOlv79/XTlxVe5baRP7svbpivNd5+RAxmYmzPjr6L+yoM8CPjnxCa8feh2t1Jo0JkVpKS8vLyIiIgCwt7enf//+dVVUG7Jt2zYGDRrE4MGDAXB1dcXc3LzBbZ988klef/31K0b+qXLfSpvQpGqwt7InwiPC1KFgJsx4YdQLWJtb88XpL7hYeZG/j/m7uuNaabF/HvwnZwrOGHWf/Vz68eyIZw3ePjk5mWPHjjFy5Eiio6N5//33WbNmDZGRkfzrX//C2dmZs2fPIoTgxhtvJDc3l0WLFvHHP/4RuLLc98aNG/Hx8alLJJd1tHLfqgfRBVwuzjfeZ7zRivO11uXRTY8PfZxN5zfxu52/o7y63NRhKUqLlJaWMn/+fN555x0cHBx4+OGHSUxMJCYmBi8vL55++mkAampq2LNnD19++SV79uxhw4YN7NixA9CV+46MjKS8vJxXX32Vl19++ZrjNPTNv8OX+xZCzADeBcyBT6SUK65aL/TrbwLKgXullEebaiuEcAG+BgKBZGChlPKiEMIS+ASI0Me3Rkr5WuteZtcWkxvDxcqLRpn7wZiEEDw46EFcbFx4ef/LPLjtQVZOXYmTjZOpQ1M6mev5pm9s1dXVzJ8/n8WLFzNv3jwAPDw86tY/+OCDzJ49G9B9s584cWLdt/ebbrqJo0ePMnXq1LrtExMTSUpKqus9pKenExERwcGDBztfuW8hhDmwEpgJhAF3CCHCrtpsJhCq/1kGfGBA2+eAHVLKUGCH/jnAAsBaShkODAMeEkIEtvQFdgeaVA0WZhaM8x5n6lAaNL/PfN6a9BZnCs6w5OclpBWnNd9IUToAKSVLly6lf//+PPXUU3XLs7Ky6h5v2LChrgT4jTfeSGxsLOXl5dTU1LBr1y7Cwq78uAwPDycnJ4fk5GSSk5Px9fXl6NGjeHp6drhy34acYhoBJEgpz0spq4C1wJyrtpmD7pu+lFLuB5yEEF7NtJ0DrNY/Xg3cqn8sAVshhAXQA6gCilv06roBqS/ON9JzZJsV5zOGqf5T+b9p/0dBRQF3br6To9lHTR2SojQrOjqazz//nJ07d14xpPWPf/wj4eHhDBo0CI1Gw9tvvw2As7MzTz31FMOHD2fIkCFEREQwa9YsQHcNormq0wMGDGDhwoWEhYUxY8YMVq5cWXeR+4MPPuCBBx4gJCSE4OBgZs6cCcDSpUvJz88nJCSEt956ixUrjDi5UlNDnPTntm5Dd2ro8vMlwPtXbfMTMK7e8x1AZFNtgcKr9nFR/9sSXSLJBcqAZY3EtQw4DBz29/dvcFhXd5B4MVEOXDVQrj291tShGCS5KFnOWj9LDl0zVG5M2GjqcJQOrCMMc+1K2mqYa0N9lauvijS2jSFtrzYCqAW8gSDgaSFE72t2IuVHUspIKWVkr169mtll17UzbScAE/0mmjgSwwQ4BPDlTV8y1H0oz+95nveOvqeGwSpKB2VIgkgH/Oo99wUyDdymqbbZ+tNQ6H/n6JffCWyRUlZLKXOAaHS9EaUB7V2czxgcrR358IYPmRc6j49PfMzTUU9TVl1m6rAURbmKIQniEBAqhAgSQlgBi4CNV22zEbhb6IwCiqSUWc203Qjco398D/CD/nEqMEW/L1tgFGDcAdBdRN6lPE7knjD5zXEtYWluyUujX+KZyGfQpGm4Y9MdnC86b+qwlA5GqnItRtHS97HZBCGlrAEeA7YCp4F1UspTQojlQojl+s02A+eBBOBj4JGm2urbrACmCSHOAdP0z0E36skOOIkuwXwmpYxt0avr4qLSopDITpkgQDcM9p4B9/Dx9I8pqizijp/u4JeUX0wdltJB2NjYkJ+fr5JEK0kpyc/Px8bG5rrbqjmpO7FHdzxKYmEiP8/7udNP1HOh7AJPRz1NbF4s9w28j8eHPo6FmbrRvzurrq4mPT29yWJ3imFsbGzw9fXF0vLKG2mbm5Na/Q/spMqry9mfuZ+FfRd2+uQA4GnryWczPmPFwRV8dvIzYnNjWTF+Rae6tqIYl6WlJUFBQaYOo1tTpTY6qX2Z+6jSVnXa00sNsTK34sXRL/KPcf8gLj+O2368rW4KVUVR2p9KEJ3UzrSd2FvZM9RjqKlDMbqbg29m3ex1eNt687jmcf5x4B9U1laaOixF6XZUguiEarQ17E7fzQTfCR2mOJ+xBToG8sVNX3BX/7v46sxXLN60mPOFapSTorQnlSA6oZicGAorC7vU6aWGWJlb8eyIZ1k5dSU55Tks+HEBq0+tplZba+rQFKVbUAmiE9KkabA0s2ScT8cszmdsE3wnsH7Oesb4jOHNw29y/9b7VcE/RWkHKkF0MlJfnG+E1whsLW1NHU67cevhxnuT3+PVca9y7uI55v84n6/PfK3GyCtKG1IJopM5X3SetJI0pvhNMXUo7U4IwS3Bt7B+znqGug/llQOv8OAvD6rehKK0EZUgOhlNmm7Y50TfzlGcry142nry4Q0f8sKoFziVd4q5G+fyyYlPqNZWmzo0RelSVILoZDSpGga4DsDD1qP5jbswIQQL+y7k+znfM95nPO8efZfbf7qd47nHTR2aonQZKkF0IrnlucTmxXb50UvXw8PWg7cnv817k9+juLKYJZuX8Mr+VyipKjF1aIrS6akE0YlEpUcBdLi5pzuCyf6T+eHWH7iz/52si1/H7A2z2XBug5prQlFaQSWITkSTqsHHzodQp1BTh9Ih2Vra8tyI5/hq9lf42fvx4t4XWbxpMbG5qhiworSEShCdRHl1OQeyDjDZb3KrivPll1ayUpNAWWWNEaPrWAa4DuDzmZ/zj3H/ILs8m8WbF/OXPX8h71KeqUNTlE5FJYhOYm/mXqq0VUzxb93w1re3n+WNrfE89PkRqmq67ukXIQQ3B9/Mj3N/5P6B97MpaROzN8zmkxOfcKnmkqnDU5ROQSWITkKTpsHByoGh7i0vzldcUc2GoxkA7EnI45lvjqPVdu0bzWwtbXly2JN8P+d7hnsM592j79Zdn1AlOxSlaSpBdAI12hp2pe9igu+EVk2is+5QGmVVtfz42Dj+cGNfNh7P5O+b4rrF3cgBDgH8e+q/WTVjFZ49PXlx74vc9uNt7Erb1S1ev6K0hEoQncCxnGMUVRa1anhrrVayel8ywwOdCfd15JFJwdw3NpDPopNZqUkwYrQd2zCPYXxx0xe8NektqrXVPLbzMe7beh8xOTGmDk1ROhyVIDqBy8X5xvqMbfE+tp/OJq3gEveN1c3QJYTghVlhzB3qw5vbzvJ/uxKNFW6HJ4RgWsA0NszZwF9G/oWkoiSW/LyE5duXcyL3hKnDU5QOQyWIDk5KSVRaFCO9RraqON9n0Un4OPVgethvd2CbmQneuG0Qswd58drPZ/h4d/eab8HSzJLb+93Oz/N+5vcRv+dU3inu3Hwnj+54lFP5p0wdnqKYnEoQHVxiYSJpJWmtOr10KrOI/ecLuHt0ABbmV/6TW5ib8c7tQ5gV7sWrm0/zya/dK0kA9LTsydLwpWyZv4UnIp7geO5xFv20iN/t+B1x+XGmDk9RTEYliA7ucnG+SX6TWryPVdHJ9LA0Z9Fw/wbXW5ib8c6iIcwc6Mkrm053u57EZbaWtjwQ/gBb5m3hsSGPcSTnCLf/dDuPbH+EI9lH1MVspdtRCaKD06RpGOg6EPee7i1qn1dayQ8xmcwf5oNjz8anJ7U0N+O9O4bW9STe2Hqm234g2lnZ8dDgh9g6fyu/G/o7TuWf4t4t93L3z3cTlRalynco3YZKEB1YTnkOJ/JOtKr20v8OpFJVq+XeMUHNbns5Sdwxwo+VmkT+8v1Jarv4fRJNsbeyZ9mgZWyZv4XnRz5PTnkOv9v5O+ZvnM+PiT+q8uJKl6cSRAcWlRYF0OLrD1U1Wj7fn8LEPr0IcbczqI25meAfc8N5eFIwXx5I5Ym1x7r0HdeG6GHRgzv63cFP837itfGvAfD8nueZtX4Wq06uoqiyyMQRKkrbUAmiA4tKi8LXzpcQp5AWtd90IpPckkruH9d876E+IQTPzujHczP78VNsFvd8epCicvVt2dLMktm9Z7P+lvWsnLoSHzsf/nXkX0z7dhqv7H+F80Xd89qN0nWpBNFB1RXn829ZcT4pJZ/uSSa4ly0TQt1aFMPyicG8tXAwh1MKmPtBNCn5ZS3aT1cjhGCC7wQ+m/EZ39z8DdMDprP+3HrmfD+H5duXsydjj7pOoXQJKkF0UNGZ0VRpq1p8eulIykVOZBRx39igVlV/nRfhyxdLR1JQVsXc/+zlcHJBi/fVFfVz6ccr417hl9t+4dEhjxJfEM/D2x/m1h9u5cvTX1JcVWzqEBWlxVSC6KA0qRocrR1bXJzv0+gkHGwsmBfh0+pYRvZ2ZcMjY3HsYcmdnxxg/dH0Vu+zq3Ht4crywcvZNn8b/xj3D2wtbFlxcAVT103lL3v+QmxubLcdFaZ0XipBdEA12hp2Z+xmgk/LivOlXyxny8kL3DHSn55WLS/uV1+Qmy3rHx5DhL8TT607zos/nOz2F68bYmluyc3BN/PV7K/4evbXzA6ezbaUbSzevJgFPy7g6zNfU1pVauowFcUgKkF0QHXF+Vo4vPXzfSkIIbh7dKBR43K2teKLpSN5YFwQa/alcMfH+8kurjDqMbqSMNcw/jr6r+xcsJMXRr2AEIJXDrzClG+m8NLel4jJiVG9CqVDMyhBCCFmCCHihRAJQojnGlgvhBDv6dfHCiEimmsrhHARQvwihDin/+1cb90gIcQ+IcQpIcQJIYRNa19oZ6JJ02BlZsVY7+svzldeVcNXB1OZMcATH6ceRo/NwtyMv8wO4993DOV0VjGz3tvDgfP5Rj9OV2JnZcfCvgtZN3sd/7vpf8wInMHmpM0s+XkJt3x/C5+c+IQLZRdMHaaiXKPZBCGEMAdWAjOBMOAOIUTYVZvNBEL1P8uADwxo+xywQ0oZCuzQP0cIYQF8ASyXUg4AJgHdZoyllBJNqoaRXiPpadnzutt/dzSD4ooa7hsbaPzg6rl5sDffPzoWexsL7vh4P+9sP0tNrTrl1BQhBOG9wnl57MvsXLCTl8e8jGsPV949+i7Tv53Osm3L2HR+k5rxTukwDOlBjAASpJTnpZRVwFpgzlXbzAHWSJ39gJMQwquZtnOA1frHq4Fb9Y+nA7FSyuMAUsp8KWW3mforoTCB9NL0Fp1e0molq6KTGOTryLAA5+YbtFIfD3s2PjaWWwZ78872c9z58QEyCtWHmyHsrOyYGzqXVTNWsXnuZh4a/BCpJak89+tzTFmnOwV16MIhNVxWMSlDEoQPkFbvebp+mSHbNNXWQ0qZBaD/fbnYUB9ACiG2CiGOCiH+2FBQQohlQojDQojDubm5BryMzuFycb6JvhOvu+3uc7kk5pZx39jAVg1tvR72Npa8s2goby0czKnMIma+s5ufT2S1y7G7Cj8HPx4d8iib523m0xs/ZYr/FDYnbeb+rfcz7dtpvHHoDU7mnVTXK5R2Z8gQl4Y+aa7+S21sG0PaNhTTOGA4UA7sEEIckVLuuGInUn4EfAQQGRnZZf7naFI1hLuFt6g432fRyfSyt2ZWuHcbRNa0eRG+RPg788TaYzz85VHmR/jy4s1hOPZovECgciUzYcZwz+EM9xzOn0f+mV3pu9ictJn/nfkfa+LW4G/vz4ygGcwMnEmIc8vurleU62FIDyId8Kv33BfINHCbptpm609Dof+dU29fu6SUeVLKcmAzEEE3kFOew8n8ky26OS4hp5RdZ3NZMioAKwvTDE4LdLPlm+VjeGxyCN/HZDD97V1ozuQ031C5Rk/LnswMmsm/p/ybqIVRvDzmZbztvPnkxCfM3TiXeRvn8XHsxyQXJZs6VKULM+ST5BAQKoQIEkJYAYuAjVdtsxG4Wz+aaRRQpD9t1FTbjcA9+sf3AD/oH28FBgkheuovWE8EusWsLa0pzrdqbxJWFmbcObLhOR/ai5WFGc/c2JcNj4zBqYcV9606xDPfHKfoUrcZZ2B0jtaOzA2dy8fTP2bHgh38acSfsLWw5b1j73Hz9zcz94e5vH/sfeIL4tVpKMWohCF/UEKIm4B3AHPgUynlq0KI5QBSyg+F7oT3+8AMdKeF7pNSHm6srX65K7AO8AdSgQVSygL9uruAP6E7HbVZStngdYjLIiMj5eHDh6/vlXdAD29/mJTiFDbN3XRd1xCKyqsZ9doOZg/y4o0Fg9swwutTWVPLv3ck8MGuRNzsrHjp5gHMGOjZbtdHuroLZRfYkbqDHak7OJJ9BK3U4mvnyw0BNzDVfyqDeg3CTKhbnZTG6U/fRza6vit84+gKCaKsuozxa8dzR787+MPwP1xX2//blchrP59h8+PjCfN2aKMIW+5EehHPfhdLXFYxk/r24uVbBuLvev1DeJXG5V/KJyotiu2p29mftZ8abQ3uPdyZ7D+ZyX6TGe45HCtzK1OHqXQwKkF0EtuSt/H0rqf59MZPGe453OB2NbVaJryuwd+1J2uXjW7DCFunplbL6n0pvLUtnhqt5LHJISyb2BtrC3NTh9bllFSVsDt9N9tTtrMnYw8VtRX0sOjBGO8xTPSdyHjf8bj1aFmFX6VraS5BGKdQj9JqmrSWFefbFpdNZlEFL90yoI0iMw4LczOWjgtiVrgXL/90in/9cpYNMRm8MDuMyX1bNp2q0jB7K3tm9Z7FrN6zqKip4OCFg+xO382u9F3sSNUNBhzoOpAJfhOY6DuR/i791Wk/pUGqB9EBVGurmfT1JCb5TeLVca9eV9vbPthLdkkFUc9Mxtys8/wnj4rP4aWNp0jOL2dCn178+ab+9PW0N3VYXZqUkrMXz9Yli9jcWCQS9x7ujPcdzxjvMYz0GomjtaOpQ1XaiepBdALHso9RXFV83aOXYtMLOZxykRdmh3Wq5AAwqa872550Y82+ZN7bcY6Z7+7mjhH+PDWtD6521qYOr0sSQtDXpS99Xfry4KAHKagoYE/GHnal7WJr8la+O/cdZsKMgW4DGeM9hrHeYxnoNrBFFYWVrkH1IDqAfx78J+vi1/Hrol+vq/7Sk1/HsO3UBfY9PxUHm857Q9rFsire2X6WLw6k0tPSnOWTgrlvbKDRSpUrzavR1nAi7wR7M/eyN3MvJ/NOopVa7CztGOk1kjHeYxjjPQZfe19Th6oYkbpI3cFJKZm5fibBTsGsnLrS4HY5xRWM/edOFo8M6PDXHwyVkFPKa5tPs+NMDm521jw6OZg7R/qrC9kmUFRZxIGsA3UJI6tMVz7F396fUV6jGO41nOEew3Ht4WriSJXWUKeYOrhzhefIKM1gafjS62r3xf4UarSSe8cEtk1gJhDibsd/7x3OkZQC3tgaz99+jOPj3ed5fGoo84f5YmmuxvS3F0drR6YHTmd64HSklCQXJ9cli01Jm1h3dh0AIU4hDPcczgjPEUR6ROJk42TawBWjUj0IE/u/4//H+zHvs3PBTnr17GVQm4rqWsau2MlQfyc+ucfwIbGdiZSS6IR83tgWz/G0QgJde/Lo5BBuHeqjEoWJ1WhriMuP4+CFgxy6cIhjOcfqSpT3de5blzCGeQ7Dwarj3Zej/EadYurgFv20CHNhzpezvjS4zbrDafzx21i+fGAkY0O69nh2KSXbT+fw1i9nOZ1VjI9TD5ZP7M2CSD9sLNWpp46guraak/knOZh1kEPZh4jJiaGythKBoJ9LP4a6D2Wox1Ai3CNaVIRSaTsqQXRg2WXZ3PDtDTwR8QQPhD9gUBspJTe9twetVrLl9+O7zfh1KSWa+Bze35nA0dRC3OyseWB8EHeNCsDOWp0p7UiqaquIzY3l0IVDHMk+QmxebF0Pw8fOhwj3iLqEEeQYpMqBmJC6BtGB7UrfBVxfcb795ws4nVXMinnh3SY5gG6I5pR+Hkzu687+8wX8JyqBFT+f4YOoRBaP9Ofu0YF4OnarmWk7LCtzKyI9I4n01H3uVGuriS+I52j2UWJyY4jOjObH8z8CumsdQ3v91sMIcw1TJUE6ENWDMKHl25eTVpzGT3N/MvjDftmawxxKLmDfn6Z2+1MsMWmFfBCVwLa4bMyFYPYgL5aO6024r7rRqyOTUpJWksbRnKMcyznG0eyjJBcnA2BpZkk/l36Eu4UT3iucwW6D8bX37VZfhtqTOsXUQbWkOF9qfjkT39TwyKRg/nBjvzaOsPNIzS9n1d5k1h1Oo7SyhhGBLtw/LpBpYZ6d7gbC7qqgooBjOceIzY0lNjeWU/mn6k5LOVs7M9BtYF3CGNhroLr4bSTqFFMHtSdjD9Xa6us6vbR6XzLmQrBkVGDbBdYJ+bv25MWbw3hyWijrDqezam8Sy784iq9zD+4Y4c+CSF/c7dXpp47MxcaFqf5Tmeo/FdCNlEosTCQ2L5YTuSc4kXeCPRl7kPoJKQMdAhnUaxDhbuEMcB1AH5c+WJurO/CNTfUgTOS5X58jOiMazUKNQaUMSitrGP2PHUzu5857d1xfQb/uplYr+SXuAp/vTyE6IZ/+5uk87n4c38hZDBg1EzM1TLZTKq0q5WT+SU7kntD1NPJiKagoAMBCWBDsFEyYa1jdTx/nPthYqC8GTVE9iA6oWlvN7vTdTPabbHCdm28Pp1FSWcP944LaOLrOz9xMMGOgFzMGenHhwDe4bP0rVhcvwS9fkr7di4yg2widvgwXT9POvqdcHzsrO0Z5jWKU1yhAdy0jsyyTuPy4uh9NmoYNCRsAMBfm1ySNvs59VdK4DipBmMCx7GOUVJUwxW+KQdtrtZJVe5MZ6u/EED+ntg2uq9BqYffreEa9Bt4RVM75iJMHtmNz4ktGnv83NR+sJMZ2FNqhdzFw4gKsrNTImc5GCIGPnQ8+dj5MC5gG6JJGVlkWp/NPcyr/FHEFcexO3833Cd8DuqTR26k3/V3608e5j654oXNfnG2cTfhKOi6VIExAk6bB2tya0d6GTfCjic8hOb+cp6f3bePIuojKEtiwHM78BIPvhNlvY21pw7BbQuGWh0mKP06m5mP6XvgRt+hHyI1+nnPuM+g1Zgkhg8YgzNQpqM5KCIG3nTfedt5MDdBdz5BSkl2erUsY+p7G3sy9bEzcWNfOvYc7fVz60Ne5b13S8Hfw7/aVbNU1iHZ2uThfiFMI709936A2iz/ZT2JOGb8+O1mVmWhOwXn46k7IOwvTX4FRD0MjQyRrqiqJ+/U75NEv6F+6HytRS4qZH9kBN+M/6R48A9RIsa4s/1I+Zy+e5ezFs8QXxBN/MZ7zReep0dYAYG1uTbBTcF3S6OPchz7OfbrUfBnqGkQHc/biWTJKMwy+czr+QgnRCfn8cUZflRyak7gTvrlP9/iu7yC46RFiFlbWDJp6J0y9k+L8HI7u/Bz7s+sZkfQfSPoPZyzDKAi+ldDJd9HLw6cdXoDSnlx7uDK6x+grevLVtdWcLzpP/MX4uqQRlRZVd10DwL2nOyFOIQQ7Bdf9DnYMxs7KzgSvom2pBNHONGkaBIJJfpMM2v6z6CRsLM24Y7i6oNooKWH/f2DbX6BXP1j0Jbj0vq5dOLi6M2rB08DTpJ2PJ/3XNXilbGTMmX9Qc3oFsdaDuRQyi9CJi3Dx8Gub16GYnKW5Zd2kSgTrlkkpybuUV5c0EgsTSShM4Jv4b6iorahr62XrdUXSCHEKobdj7+ua46WjUaeY2tntP92OhZkFX97UfHG+grIqRr+2g3kRvrw2L7wdouuEqi/Bj7+H2LXQ/2a49UOwNtI3OSlJPX2QrL1f4Z2xFT+ZiVYK4m3CKek9i8Bxi3D3CTTOsZROp1ZbS2ZpJucKz9UljcTCRJKKkqjSVtVt52Pno+tl6JNGoEMggY6BHeJmP3WKqQO5UHaBuPw4noh4wqDtvzqYSmWNlvvGBrZtYJ1VUQZ8vRgyj8HkP8P4Z8CYF5iFwD9sJP5hI5FaLYlxh7mw72u8s7bR//RrcPo1Tlv0J99/Bn6j5uIfOkiVhOhGzM3M8XPww8/Bjyn+v41IrNHWkF6SfkXSOFd4jr2Ze+uubwC49XAj0CGQIMegut9BjkF42XphbtYxyuioBNGOdqXpivMZMry1ulbLmn3JjA91o4+HfVuH1vmkHoCv74Lqclj0P+g3q00PJ8zMCB44guCBI5BSknI2hqx963BP+5lx59+G82+TKnzI8piI09BbCBl2A+YWnXcaWKXlLMwsCHTU9RIuj6QC3f1P6SXpJBclk1ScRFJREslFyWxN3kpxVXHddlZmVgQ4BhDkEESgoz5x6B/bWtq272tp16N1c5o0DQEOAQQ5Nn+z2+YTWWQXV6pTSw05sho2PQ2OvnDPRnDv366HF0IQ0HcoAX2HAq+RkxpP8t712CT/wtCsr7G68D+KfrbjnMMo6DOD4NG34uxq2GRQStdlaWZZ10uYzG8DKKSUXKy8qEscRUkkF+t+nyk4w/bU7Wiltm5b9x7uBDoGEuAQUPcT4hTSZnOFq2sQ7aS0qpTxX49ncb/FPDP8mWa3v3VlNEWXqtnx1ETMVME5ndpq2PInOPQxBE+B+f+Fni6mjuoKRYUFxEf/AGe3EFq0F2eKqZFmxFv1p8h7Am6DZxIyaCxmFuq7mdK8qtoq0krSrkgcycXJpBanUlhZCMC0gGm8NemtFu1fXYPoIPZk7qFGW8Nk/+aL8x1NvUhMWiF/u2WASg6XleXBunsgZQ+M+R1MfQnMO96fr6OTCyNm3Qez7qO2poazMbsojNmIy4U9jEn5AFI+oHCjPUkOw9H2noz/8Nn08rm+EVdK92FlblV3gftqhRWFpJSkYGXWdlUAOt7/sC5Kk6rB2dqZIb2GNLvtZ9HJ2NtYcNuwtuk2djpZsbD2TijNgbkfweDbTR2RQcwtLOgTORUideehC7LTOX9wE7XndtC7+AC9YnZCzAskm/mR5Toaq743EBI5DUenjtUrUjomJxsnnGyc2vQYKkG0g2ptNb9m/MoUvynNjk7IKrrE5hNZ3DcmEFs1lSac/A6+f1R3Kun+LeATYeqIWszFwxeXmx8CHkJbq+Vc3CHyj2/GLn03ETkbsM5dR82vZsRbhnLRfSR2fScREnkDNrZd585dpXNRn0Dt4Gj2UUqqSgw6vfT5vhSklNwzJrDtA+vItLWw8xXY8xb4jYKFa8Dew9RRGY2ZuRmh4SMJDR8JQNWlUs4c01AUtxOH7P0My/gSy8w11Ow046xVHwrcR9IjdCK9I6Zi7+Bk2uCVbkMliHZQV5zPq+nifJeqavnfwVSmhXng59J5775stYoi+O4BOLcNht0LM98Ai65dbdWqhx39xtwMY24GoLSkiFNHdlAWH4VL7kGGpX+BZcZqqjXmnLEMocA1ApveYwgYMhlXdWe30kYMShBCiBnAu4A58ImUcsVV64V+/U1AOXCvlPJoU22FEC7A10AgkAwslFJerLdPfyAOeElK+WbLX6JpSSnRpGoY5TWq2Vvuv4/JoLC8mvvHduM5H3LPwto74GIyzHoLhi81dUQmYWfvyJBJ82DSPADKS4s4e3QnZfEaHHIPM+zCt1hnfwX7IF14keM0GK3vSNwHTMQ3dAhm5h3jRiulc2s2QQghzIGVwDQgHTgkhNgopYyrt9lMIFT/MxL4ABjZTNvngB1SyhVCiOf0z5+tt8+3gZ9b+wJN7ezFs2SWZbJs0LImt5NS8ll0EmFeDowI6qYXKc9u1fUczK3g7o0QONbUEXUYPe0cGTBhLkyYC0BVxSXOxEZzMf5XrLMOEXhxLy4Xt8AJKMaWJJswSj0isQseQ2D4WBydXU38CpTOyJAexAggQUp5HkAIsRaYg+7b/WVzgDVSd1PFfiGEkxDCC13voLG2c4BJ+vargSj0CUIIcStwHihr+UvrGHam7UQgmOg3scntohPyOZtdypsLBne/cg1S6q417Pg7eIbr7ox2UqdNmmJl04N+I26AETcAoK3VkpJ4guxTuyFtPx6FxxmsH1ar3SFIMfch134A0mcYrn1G499/OBbWPUz8KpSOzpAE4QOk1Xuejq6X0Nw2Ps209ZBSZgFIKbOEEO4AQghbdIliGtDoHWVCiGXAMgB//45b6VSTqmFQr0G49XBrcrtPo5Nws7Pi5sFe7RRZB1FVBj88BqfWw8D5cMv7YNWNr7+0kJm5GQF9BhPQZzDwOwBKC3NJjf2VkvMHsMmOIahoP65FWyEOqjZYcM6yNwVOAxG+w3DrOwb/0EFYqBv4lHoM+Wto6Ovs1bdfN7aNIW2v9jfgbSllaVPfpKWUHwEfge5O6mb2aRIXyi5wuuA0v4/4fZPbJeWVsfNMDk9MDcXaohudO76YAmsXQ/ZJuOFvMPaJRif3Ua6fnVMvwibMgwm66xhSqyUjNYGMU3uoTj2C48VYBuZuxjZvPcRAqexBilVvSpzCMPMeTK/Q4fj1GYqFlbVpX4hiMoYkiHSgfn/fF8g0cBurJtpmCyG89L0HLyBHv3wkcJsQ4nXACdAKISqklIZNv9aBRKVFATQ7vHVVdBKW5oLFozpuT8jokvfAuruhtgYWfwOh00wdUZcnzMzwCeyDT2Af4H4AamtqSEmIJT9+L7XpR3AoPM2gnI30zP0GjkOVtCDRMpCLDv2QnoOwDxyGX/9IbO2dTPpalPZhSII4BIQKIYKADGARcOdV22wEHtNfYxgJFOk/+HObaLsRuAdYof/9A4CUcvzlnQohXgJKO2NygHrF+RwaH5VUdKmab46kc/Mgb9ztbdoxOhOREg59Alue003qs+grcAsxdVTdlrmFBQH9Igjo99sNiNqaGlIST5Jz9iDV6THYXYwjuGAXzgU/QRxoNwlSzbzJtu1DlWsYPXzD8QyNwMs/VM3n3cU0myCklDVCiMeAreiGqn4qpTwlhFiuX/8hsBndENcEdMNc72uqrX7XK4B1QoilQCqwwKivzMRKqko4eOEgd/W/q8mLzt8cTqO8qpb7usPQ1ppKXRXWY59Dnxkw7yOwUXcJdzRmFhYE9B1CQN8hdcukVktWeiLZ8YeoSDtGj/xT+JWdxLNUAylAtO4UVYZVIMUOfRDu/bEPGIxXn2E4uHSdGxy7G1XNtY1sSdrCH3b/gdUzVhPh0XB5iFqtZOIbGrwcbfhm+Zh2jrCdlVyAr5dA+kHdxD6T/2zcyX0UkygvLiA9/giFycfRZp/CoegsPlVJOIrfBiDm4swF6yBKHfsgPMJwCBiET/BgHJ276XDuDkRVczWRnWk7cbFxYXCvwY1u80tcNukXL/Hnm9p3PoN2l3EE1t4FFYWwYBUMmGvqiBQj6engQp/h02D4b9eQpFZLRnoyuYnHKE8/gUXeGZzLzhGa/R02OWvhhG67bFzJsfan3CEY0asvDn4D8AoehGMvXzVYoYNQCaINVGur2ZO+h6kBU5sszvdZdBI+Tj2YFtaFu+AxX8GPT4CdByzdprvPQenShJkZPv698fHvDcyvW66tqSEr9Qx552MozzyNef45HMqSCMrZhF3ut3V3VhVjS5alH8W2QdS6hGLl1R/XgAF4BvbDWo2oalcqQbSBI9lHKKkuYbJf46OXTmUWcSCpgOdv6oeFeRc81VJbA7+8CPtXQuB4WLAabNXdvN2ZmYUFXr0H4tV74BXLtbVaMjOTyDkfS1lGHGZ557ArTSKwcD+9Cn/W3TIbDZXSgmQzTwps/Ki0D0S4BWPv1Qe3wDDcfXojOsg8zl2JShBtQJOqL87n3Xhxvs+ik+lhac7tkV1waGt5AXx7H5yPghEPwY2vgrman1lpmJm5Gd5+wXj7BQNXnn4sLswjO/EExekn0WbHY1mcjHN5Kl7lh7HJqa7rdVRISy6Ye1HYw59KhwDMXIPp4dUHV/8wPLwDVW2qFlIJwsiklGjSNIz2Gk0Pi4ZLGeSWVLIxJpPbh/vh2LOLfXBmx+mK7RVn6u6Kjlhi6oiUTszByQ2HYZNh2JW9cW1tLRcyk8lLiaM0M57avERsSpJxKU/Bu/QA1lnVcFK37SVpRaa5F4XWvlTa+4NzID09euPs0wcP/1Bsetia4JV1DipBGFn8xXiyyrJYPnh5o9v870AqVbVa7h0b2H6BtYfTP8L6h8DaDu7dBH4jTB2R0kWZmZvj6ReMp18wcPMV62pqashMTyQ/7QyXLpxF5idiU5xMr4oU3MsP6noe8b9tn4sLeZZelNv6UusYgIVrILYewTj79MHNK6Bb9z5UgjAyTaoGgWCC74QG11fW1PL5/hQm9e1FcC+7do6ujWi1sOufsGsF+AyD278AB29TR6V0UxYWFngH9sU7sO8166S2lvycDHJT4ym9kEBVXhIWRSn0LM/Ap/AI7he3YZby29D/KmnBBTN3Cq28KbP1pdbRHyvXAOzcg3D16Y2bpz/mXTiBqARhZJo0DYN7DW60ON+m2CzySiu7zo1xlSWwYTmc+QkG3wmz3wbLbnBHuNIpCTNzXD39cfX0R1cP9Erl5WXkpidwMTOBipzzcDEFq5JUHCoy8C84g1NBKST9tn2VNOeCmRuFlh5c6ulFrb0fFs5+9OgViKNnIG6+wVj3sG+/F2hkKkEY0eXifE8Oe7LB9VJKPo1OIsTdjgmhTVd37RTyE3XF9vLOwowVMHK5Gr+udGo9e9rWq4p7rYqSi+RmJFCYlcSl3GS0hWlYlGRgW5GFb+ERel3chnnalTcfX8SBfPNelFh7UmHrjdbBD0sXP2x7BeDsEYCblz9WVh1zxkSVIIxIk6YBaHR46+GUi5zMKOaVWwd2/jkfEnboRioJM1iyHnpPMnVEitLmbOyd8es3HL9+wxtcX1FRQUZGMkXZSZTnJFF7MQ3zkgxsyjNxqUilV/lheuZWQuJvbWqlIEc4U2jhRqm1B9U9PcDBG0tnX3q6+eHkGYiLRwBWJriYrhKEEWlSNQQ6BBLk2PDpo8+ik3DsYcm8CJ92jsyIpIR9K+GXF6BXP93kPi5d5HSZorSSjY0N/sH9ILhfwxtISWlRPgWZCRRnp1CRn0ZtUQZmJRewuXQBl0vJuJYexj730jVNC7GnwNyVUit3Knp4UGvnjbmjN05BQ+gTMalNXo9KEEZSUlXCoexDLOnf8LDO9IvlbDl5gQcn9KanVSd926sv6e6Kjv0a+t8Ct36gG7GkKIphhMDOyQ07JzcIG9XgJlJKiosvkp+ZQlGOLonUFGZiUZaJdXk2dlU5eF86i1tBIQBHkqeAShAd256MPdRoaxqd++HzfSkIIbh7dGD7BmYsRRnw9WLIPKYrtDf+GVVsT1HagBACB0cXHBxdoP/QRrerrLxEflYqbTkHpUoQRqJJ1eBi48Igt0HXrCuvquGrg6nMGOCJj1MnnAc4db+uEmt1ue6UUr9Zpo5IUbo9a+seDQ7lNSb1FdAIqmur2ZOxh4m+Exsszvfd0QyKK2q4f1xg+wfXWkdWwarZulNJD2xXyUFRuhHVgzCCw9mHGy3Op9VKPotOYpCvIxH+ziaIroVqq3Wzvh36BIKnwG2fQo9OFL+iKK2mehBGoEnTYGNuwyjvay867T6Xy/ncMu4fG9R5hraW5sKaObrkMOZxWPytSg6K0g2pHkQrXS7ON8p7VIPF+T6NTsbd3pqbwtvyUpIRZR3X3fxWlgvzPoZBC00dkaIoJqJ6EK10puAMF8ouMMVvyjXrEnJK2H02lyWjArCy6ARv9cnv4L83gtTC/VtUclCUbk71IFpJk9Z4cb7PopOxsjDjzpEdfM4HbS3s/DvseRv8RsHtn4Odu6mjUhTFxFSCaKWotCiGuA/BtceVs6UVllex/mgGtw7xxtWuA0+TeKkQvnsAEn6BYffCzDfAomPWhVEUpX11gvMeHVdWaRanC043OHpp7aE0LlXXduyqrbln4ZOpcF4Ds96Cm99VyUFRlDqqB9EKl4vzTfKbdMXymlota/YmM7q3K/29HEwQmQHObtX1HMyt4O6NEDjW1BEpitLBqB5EK2jSGi7Ot/VUNplFFdzXEWeMkxJ2vwn/u11XZG9ZlEoOiqI0SPUgWqi4qpjDFw6zZMC1xfk+i07C36UnU/t7mCCyJlSVwQ+PwqkNMPA2uOXfYNXT1FEpitJBqQTRQnvS91Aja64Z3hqbXsjhlIu8MDsMc7MOdGPcxRTd/Q3ZJ+GGv8HYJ9TkPoqiNEkliBaKSovCxcaFcLfwK5Z/Fp2MnbUFCyN9TRNYQ5J+hXV364azLv4GQq+dalFRFOVq6hpEC1TXVvNrxq9M8pt0RXG+nOIKforN5LZhvtjbWJowQj0p4cBHurIZtm7w4E6VHBRFMZjqQbTAoexDlFaXXjO89Yv9KdRoJfeOCTRNYPXVVMKmp+HY59Bnhq5shk0HHVGlKEqHpBJEC2hSdcX5RnqNrFtWUV3LlwdSmdrPnUC39p879golF3TzN6Qf1E3sM/nPanIfRVGum0oQ10lKSVR6FKO9R19RnG/j8Uzyy6q439Q3xqUf0c38VlEEC1bBgLmmjUdRlE7LoK+VQogZQoh4IUSCEOK5BtYLIcR7+vWxQoiI5toKIVyEEL8IIc7pfzvrl08TQhwRQpzQ/762Cp4JnS44zYWyC1ecXpJS8umeJPp62DM62LWJ1m0s5iv4bCaYW8LSbSo5KIrSKs0mCCGEObASmAmEAXcIIcKu2mwmEKr/WQZ8YEDb54AdUspQYIf+OUAecLOUMhy4B/i8xa+uDWjSNJgJMyb6Taxbtv98AWculHD/uEDTzPlQWwNb/gTfLwe/EfBgFHiGN9tMURSlKYb0IEYACVLK81LKKmAtMOeqbeYAa6TOfsBJCOHVTNs5wGr949XArQBSymNSykz98lOAjRCiw1S7i0qLYkivIbjYuNQt+zQ6CeeelswZ4tP+AZUXwBfzYP9/YMRDsGQD2JqwF6MoSpdhSILwAdLqPU/XLzNkm6baekgpswD0vxuqLz0fOCalrLx6hRBimRDisBDicG5urgEvo/UySzM5U3DmitNLqfnlbD+dzeKRAdhYXjsfdZvKjoOPJ0PqPpizEm56XXd6SVEUxQgMSRANnTORBm5jSNuGDyrEAOCfwEMNrZdSfiSljJRSRvbq1cuQXbZaQ8X5Vu1NxlwIlowOaJcY6sRthE9ugOoKuHczDL2rfY+vKEqXZ8gopnTAr95zXyDTwG2smmibLYTwklJm6U9H5VzeSAjhC2wA7pZSJhryQtqDJk1DkGMQgY6BAJRUVLPucBqzBnnh4WDTPkFotbBrBez6J/gMg9u/BIdOMp2poiidiiE9iENAqBAiSAhhBSwCNl61zUbgbv1oplFAkf60UVNtN6K7CI3+9w8AQggnYBPwJylldMtfmnEVVxVz5MKRK04vfXskndLKmvab86GyBL6+S5ccBt+p6zmo5KAoShtptgchpawRQjwGbAXMgU+llKeEEMv16z8ENgM3AQlAOXBfU231u14BrBNCLAVSgQX65Y8BIcALQogX9MumSynrehim8Gv6r9TImroEodVKVu1NJsLfiSF+Tm0fQH4irL0T8s7BjBUwcrkqtqcoSpsy6EY5KeVmdEmg/rIP6z2WwKOGttUvzwemNrD8FeAVQ+JqT1FpUbjauDKo1yAAdp7JISW/nGem9237gyfsgG/vA2EGS9ZD70ltf0xFUbo9VX/BANW11ezJ2MMkv0mYCd1b9tneJLwcbZgx0LPtDiwl7P03fHkbOPjCgxqVHBRFaTeq1IYBDl3QFee7PHrpzIViohPy+eOMvliat1GOrb4EPz4BsV9D/1vg1g/A2q5tjqUoitIAlSAMsDNtJz0sejDKaxQAq6KTsbE0447h/m1zwKJ03eQ+WTG6Qnvjn1HF9hRFaXcqQTRDSklUWhSjvUZjY2FDQVkVG45lMC/CF2dbK+MfMHW/bqRSdQUs+gr63WT8YyiKohhAfS1tRlxBHNnl2Uz2141e+upgKpU1Wu4fG2j8gx1ZBatmg7U9PLBdJQdFUUxK9SCaEZUWpSvO5zuR6lota/YlMz7UjVAPe+MdpKYKtjwHh/8LwVPgtk+hh7Px9q8oitICqgfRDE2qhiG9huBs48zmE1lkF1cad86H0lz4/FZdchjzOCz+ViUHRVE6BJUgmpBRmkH8xfi6m+M+jU6mt5stE/sYqfZT1nH4aBJkHNFNCTr972DWzgX/FEVRGqESRBOi0qIAmOw/maOpFzmeVsi9YwMxMzPCHcwnvoX/3ghIuH8LDFrY+n0qiqIYkUoQTdCkaujt2JsAhwA+3ZOEvY0F8yN8W7dTbS388lf4bil4D4FlUeA91BjhKoqiGJVKEI0oqizicPZhJvtNJqvoEj+fvMCi4X7YWrfiuv6lQvjf7RD9Dgy7D+7eCHYNTYOhKIpiemoUUyP2ZOyhVtYy2X8ya/alIKXk7tGBLd9h7ln4ahEUpsCst2D4UqPFqiiK0hZUgmiEJk2Dq40rIQ5hfHVQw/QwT/xcerZsZ/FbYP2DYG4F9/wIAWOMG6yiKEobUKeYGlBVW1VXnO+HmCwKy6u5ryU3xkkJu9/U9RxcgnTXG1RyUBSlk1A9iAYcunCIsuoyJvlN4tVvkhjg7cCIIJfr20lVGXz/CMR9DwNvg1v+DVYt7IEoiqKYgOpBNECTpqGHRQ9qy0I4l1PKfWODENczOc/FFPjvdIj7Aaa9DPM/UclBUZROR/UgriKlRJOmYYz3GL7cn4WbnRU3D76OaT2TfoV1d+uGsy7+FkJvaLtgFUVR2pDqQVwlriCOnPIcBjqNZueZHBaPDMDawoC7m6WEA/8Ha+aArRs8uFMlB0VROjXVg7iKJlWDmTAjIdkPK/MiFo8yYM6HmkrY9BQc+wL6zIR5H4GNQ9sHqyiK0oZUgriKJk3DILchbNxfzOzBXrjb2zTdoOSCbv6G9EMw4Q8w6Xk1uY+iKF2C+iSrJ70knbMXz2JfO5jyqtrmq7amH9EV28s+BQtWw5S/qOSgKEqXoXoQ9Vwuzncs3ocRgS4M9HFsfOOY/8GPvwd7D1j6C3gObI8QFUVR2o36uluPJk2Dh00AWXl23D8usOGNamtgy5/g+4fBbwQ8GKWSg6IoXZLqQegVVRZxJPsITtXT8XHqwbQwz2s3Ki+Ab+6FpF0wcjlMfwXMLds9VkVRlPagEoTerxm/UitrSU0L4k9TAjG/es6H7FPw1R1QkgVzVsLQu0wTqKIoSjtRCUJPk6rBCkdqtQEsHO535cq4jbBhOVjbw72bwW+4aYJUFEVpRypBoCvO92vGHi4VDuS2Yf449tCfNtJqIeo12P06+ETC7V+Aw3XcVa0oitKJqQQBHLxwkEs15VSWhHHPmEDdwopi2PAQxG+GIYt1czhYNnNPhKIoSheiEgSwPWUnaK0Y5zOK4F52kJ8Ia++EvHMw458w8iG4nmJ9iqIoXUC3TxBaqWVb0k6qS0NZemMfSNgO394PwgyWbIDeE00doqIoikl0+/sg4vLiKKnJx1UMZXzuV/DlAnDwhQc1KjkoitKtGZQghBAzhBDxQogEIcRzDawXQoj39OtjhRARzbUVQrgIIX4RQpzT/3aut+5P+u3jhRA3tvZFNuXLkz8jpeA/VkcQv7wA/WbD0m26GeAURVG6sWYThBDCHFgJzATCgDuEEGFXbTYTCNX/LAM+MKDtc8AOKWUosEP/HP36RcAAYAbwH/1+2sSulG2EVUgGZ26FyX+BhWvA2q6tDqcoitJpGNKDGAEkSCnPSymrgLXAnKu2mQOskTr7ASchhFczbecAq/WPVwO31lu+VkpZKaVMAhL0+zG6YzE/UiIucOOlUlj0FUz8g7oYrSiKomdIgvAB0uo9T9cvM2Sbptp6SCmzAPS/3a/jeAghlgkhDgshDufm5hrwMq5VauPKmEvWDJv2IfS7qUX7UBRF6aoMGcXU0FdqaeA2hrRtyfGQUn4EfAQQGRnZ3D4bNL7fGMb3O9ySpoqiKF2eIT2IdKB+7QlfINPAbZpqm60/DYX+d851HE9RFEVpY4YkiENAqBAiSAhhhe4C8sarttkI3K0fzTQKKNKfNmqq7UbgHv3je4Af6i1fJISwFkIEobvwfbCFr09RFEVpoWZPMUkpa4QQjwFbAXPgUynlKSHEcv36D4HNwE3oLiiXA/c11Va/6xXAOiHEUiAVWKBvc0oIsQ6IA2qAR6WUtcZ6wYqiKIphhJQtOn3foURGRsrDh9W1BEVRlOshhDgipYxsbH23v5NaURRFaZhKEIqiKEqDVIJQFEVRGqQShKIoitKgLnGRWgiRC6S0YhduQJ6RwjEmFdf1UXFdHxXX9emKcQVIKXs1trJLJIjWEkIcbupKvqmouK6Piuv6qLiuT3eMS51iUhRFURqkEoSiKIrSIJUgdD4ydQCNUHFdHxXX9VFxXZ9uF5e6BqEoiqI0SPUgFEVRlAapBKEoiqI0qFsnCCHEDCFEvBAiQQjxXDscz08IoRFCnBZCnBJCPKFf/pIQIkMIEaP/ualemz/p44sXQtxYb/kwIcQJ/br3hGjdXKlCiGT9/mKEEIf1y1yEEL8IIc7pfzu3Z1xCiL713pMYIUSxEOL3pni/hBCfCiFyhBAn6y0z2vujL2//tX75ASFEYCviekMIcUYIESuE2CCEcNIvDxRCXKr3vn3YznEZ7d/NyHF9XS+mZCFEjAner8Y+G0z7Nyal7JY/6MqPJwK9ASvgOBDWxsf0AiL0j+2Bs0AY8BLwTAPbh+njsgaC9PGa69cdBEajm4HvZ2BmK2NLBtyuWvY68Jz+8XPAP9s7rqv+vS4AAaZ4v4AJQARwsi3eH+AR4EP940XA162IazpgoX/8z3pxBdbf7qr9tEdcRvt3M2ZcV63/F/CiCd6vxj4bTPo31p17ECOABCnleSllFbAWmNOWB5RSZkkpj+oflwCnaWC+7XrmAGullJVSyiR0822MELoZ+ByklPuk7l97DXBrG4Q8B1itf7y63jFMEddUIFFK2dQd820Wl5RyN1DQwPGM9f7U39e3wFRDejkNxSWl3CalrNE/3Y9uVsZGtVdcTTDp+3WZvv1C4Kum9tFGcTX22WDSv7HunCB8gLR6z9Np+sPaqPTdu6HAAf2ix/SnBD6t141sLEYf/eOrl7eGBLYJIY4IIZbpl3lI3cyA6H+7myCuyxZx5X9cU79fYNz3p66N/sO9CHA1Qoz3o/sWeVmQEOKYEGKXEGJ8vWO3V1zG+ndri/drPJAtpTxXb1m7v19XfTaY9G+sOyeIhjJnu4z5FULYAd8Bv5dSFgMfAMHAECALXTe3qRjbIvaxUsoIYCbwqBBiQhPbtmdcCN10tbcA3+gXdYT3qykticPoMQoh/oxuVsYv9YuyAH8p5VDgKeB/QgiHdozLmP9ubfFvegdXfglp9/ergc+GRjdt5DhGja07J4h0wK/ec18gs60PKoSwRPcH8KWUcj2AlDJbSlkrpdQCH6M7/dVUjOlcedqg1bFLKTP1v3OADfoYsvVd1svd6pz2jktvJnBUSpmtj9Hk75eeMd+fujZCCAvAEcNP0VxDCHEPMBtYrD/VgP50RL7+8RF05637tFdcRv53M/b7ZQHMA76uF2+7vl8NfTZg4r+x7pwgDgGhQogg/TfURcDGtjyg/nzff4HTUsq36i33qrfZXODyCIuNwCL96IMgIBQ4qO9qlgghRun3eTfwQyvishVC2F9+jO4i50n98e/Rb3ZPvWO0S1z1XPHNztTvVz3GfH/q7+s2YOflD/brJYSYATwL3CKlLK+3vJcQwlz/uLc+rvPtGJcx/92MFpfeDcAZKWXd6Zn2fL8a+2zA1H9jzV3F7so/wE3oRgskAn9uh+ONQ9eliwVi9D83AZ8DJ/TLNwJe9dr8WR9fPPVG3gCR6P6DJQLvo78rvoVx9UY3IuI4cOrye4Hu/OQO4Jz+t0t7xqXfX08gH3Cst6zd3y90CSoLqEb3TWypMd8fwAbdKbQEdKNQercirgR055ov/41dHrkyX//vexw4CtzcznEZ7d/NmHHpl68Cll+1bXu+X419Npj0b0yV2lAURVEa1J1PMSmKoihNUAlCURRFaZBKEIqiKEqDVIJQFEVRGqQShKIoitIglSAURVGUBqkEoSiKojTo/wFGXsvR0y146wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Three settings of the lrate hyperparameters.\n",
    "opts = [NoamOpt(512, 1, 4000, None),\n",
    "        NoamOpt(512, 1, 8000, None),\n",
    "        NoamOpt(256, 1, 4000, None)]\n",
    "plt.plot(np.arange(1, 20000), [[opt.rate(i)\n",
    "                                for opt in opts] for i in range(1, 20000)])\n",
    "plt.legend([\"512:4000\", \"512:8000\", \"256:4000\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06c41149",
   "metadata": {},
   "source": [
    "### 正则化\n",
    "### 标签平滑\n",
    "\n",
    "在训练过程中，我们使用的label平滑的值为$\\epsilon_{ls}=0.1$ [(cite)](https://arxiv.org/abs/1512.00567)。虽然对label进行平滑会让模型困惑，但提高了准确性和BLEU得分。\n",
    "\n",
    "> 我们使用KL div损失实现标签平滑。我们没有使用one-hot独热分布，而是创建了一个分布，该分布设定目标分布为1-smoothing，将剩余概率分配给词表中的其他单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1689b63d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LabelSmoothing(nn.Module):\n",
    "    \"Implement label smoothing.\"\n",
    "\n",
    "    def __init__(self, size, padding_idx, smoothing=0.0):\n",
    "        super(LabelSmoothing, self).__init__()\n",
    "        self.criterion = nn.KLDivLoss(reduction='sum')\n",
    "        self.padding_idx = padding_idx\n",
    "        self.confidence = 1.0 - smoothing\n",
    "        self.smoothing = smoothing\n",
    "        self.size = size\n",
    "        self.true_dist = None\n",
    "\n",
    "    def forward(self, x, target):\n",
    "        assert x.size(1) == self.size\n",
    "        true_dist = x.data.clone()\n",
    "        true_dist.fill_(self.smoothing / (self.size - 2))\n",
    "        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)\n",
    "        true_dist[:, self.padding_idx] = 0\n",
    "        mask = torch.nonzero(target.data == self.padding_idx)\n",
    "        if mask.dim() > 0:\n",
    "            true_dist.index_fill_(0, mask.squeeze(), 0.0)\n",
    "        self.true_dist = true_dist\n",
    "        return self.criterion(x, true_dist.requires_grad_(False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b499c467",
   "metadata": {},
   "source": [
    "下面我们看一个例子，看看平滑后的真实概率分布。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "2703a9a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\56550\\AppData\\Local\\Temp/ipykernel_30536/410927135.py:19: UserWarning: This overload of nonzero is deprecated:\n",
      "\tnonzero(Tensor input, *, Tensor out)\n",
      "Consider using one of the following signatures instead:\n",
      "\tnonzero(Tensor input, *, bool as_tuple) (Triggered internally at  ..\\torch\\csrc\\utils\\python_arg_parser.cpp:766.)\n",
      "  mask = torch.nonzero(target.data == self.padding_idx)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x27222a8aa60>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAADsCAYAAAB39h09AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAM/0lEQVR4nO3df6hf9X3H8edryVXnqkhJW22Sav8IMh3UurtU8Z/o5hYzWfaHjAirxY5eKgoWCsN1YLc/BvtjlE0ihnQNVlZ0A0sXuljnOjt1zNaYRWtMwy4ieElAql0006lp3/vjfiWX6/fm/vge7/nOz/MBl5zzPZ+cz4eDPv1y7vd7TFUhSfrg+6W+FyBJWh0GX5IaYfAlqREGX5IaYfAlqREGX5IasXaUv5zkw8DfAxcBLwJ/UFU/GzLuReB14OfAyaqaHGVeSdLyjfoO/w7g+1W1Cfj+YH8hV1fVZcZekvoxavC3A98cbH8T+P0RzydJep+MGvyPVdUxgMGfH11gXAH/nOTpJFMjzilJWoFF7+En+Rfg/CGH/nQZ81xVVUeTfBR4JMlPquqxBeabAqYA1rDm18/m3GVM88FV557d9xLGxsUX/bTvJYyNIy+u63sJGjP/++bPeOft/8mwYxnlWTpJjgBbqupYkguAH1TVxYv8nT8DTlTVXy12/nPz4fpMfnPF6/sgeXvrb/S9hLHx6J6v972EsXH157/Q9xI0Zg78+128fnxmaPBHvaWzF/jcYPtzwD/OH5DkV5Kc8+428NvAcyPOK0laplGD/5fAtUn+C7h2sE+SjyfZNxjzMeCJJM8APwL+qaq+N+K8kqRlGulz+FX1CvCeey5VdRTYNth+AfjUKPNIkkbnN20lqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqRGdBD/J1iRHkkwnuWPI8SS5a3D82SSXdzGvJGnpRg5+kjXA3cB1wCXAjUkumTfsOmDT4GcKuGfUeSVJy9PFO/zNwHRVvVBVbwMPANvnjdkO3FezngTOS3JBB3NLkpaoi+CvB16asz8zeG25YwBIMpVkf5L97/BWB8uTJEE3wc+Q12oFY2ZfrNpdVZNVNTnBmSMvTpI0q4vgzwAb5+xvAI6uYIwk6X3URfCfAjYl+WSSM4AdwN55Y/YCNw0+rXMFcLyqjnUwtyRpidaOeoKqOpnkNuBhYA2wp6oOJfni4PguYB+wDZgG3gBuHnVeSdLyjBx8gKrax2zU5762a852Abd2MZckaWX8pq0kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjDL4kNcLgS1IjOgl+kq1JjiSZTnLHkONbkhxPcnDwc2cX80qSlm7tqCdIsga4G7gWmAGeSrK3qp6fN/Txqrp+1PkkSSvTxTv8zcB0Vb1QVW8DDwDbOzivJKlDXQR/PfDSnP2ZwWvzXZnkmSQPJbm0g3klScsw8i0dIENeq3n7B4ALq+pEkm3Ad4BNQ0+WTAFTAGdxdgfL+2B4dM/X+17C2Lj681/oewnS/0tdvMOfATbO2d8AHJ07oKpeq6oTg+19wESSdcNOVlW7q2qyqiYnOLOD5UmSoJvgPwVsSvLJJGcAO4C9cwckOT9JBtubB/O+0sHckqQlGvmWTlWdTHIb8DCwBthTVYeSfHFwfBdwA3BLkpPAm8COqpp/20eS9D7q4h7+u7dp9s17bdec7Z3Azi7mkiStjN+0laRGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGdBL8JHuSvJzkuQWOJ8ldSaaTPJvk8i7mlSQtXVfv8O8Ftp7m+HXApsHPFHBPR/NKkpaok+BX1WPAq6cZsh24r2Y9CZyX5IIu5pYkLc1q3cNfD7w0Z39m8Np7JJlKsj/J/nd4a1UWJ0ktWK3gZ8hrNWxgVe2uqsmqmpzgzPd5WZLUjtUK/gywcc7+BuDoKs0tSWL1gr8XuGnwaZ0rgONVdWyV5pYkAWu7OEmS+4EtwLokM8BXgQmAqtoF7AO2AdPAG8DNXcwrSVq6ToJfVTcucryAW7uYS5K0Mn7TVpIaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqREGX5IaYfAlqRGdBD/JniQvJ3lugeNbkhxPcnDwc2cX80qSlm5tR+e5F9gJ3HeaMY9X1fUdzSdJWqZO3uFX1WPAq12cS5L0/ljNe/hXJnkmyUNJLl3FeSVJdHdLZzEHgAur6kSSbcB3gE3DBiaZAqYAzuLsVVre+Pudj1/W9xLGxhk81fcSpLGVemPBY6vyDr+qXquqE4PtfcBEknULjN1dVZNVNTnBmauxPElqwqoEP8n5STLY3jyY95XVmFuSNKuTWzpJ7ge2AOuSzABfBSYAqmoXcANwS5KTwJvAjqqqLuaWJC1NJ8GvqhsXOb6T2Y9tSpJ64jdtJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRBl+SGmHwJakRIwc/ycYkjyY5nORQktuHjEmSu5JMJ3k2yeWjzitJWp61HZzjJPDlqjqQ5Bzg6SSPVNXzc8ZcB2wa/HwGuGfwpyRplYz8Dr+qjlXVgcH268BhYP28YduB+2rWk8B5SS4YdW5J0tJ1eg8/yUXAp4Efzju0Hnhpzv4M7/2PwrvnmEqyP8n+d3iry+VJUtM6C36SDwEPAl+qqtfmHx7yV2rYeapqd1VNVtXkBGd2tTxJal4nwU8ywWzsv1VV3x4yZAbYOGd/A3C0i7klSUvTxad0AnwDOFxVX1tg2F7gpsGnda4AjlfVsVHnliQtXRef0rkK+Czw4yQHB699BfgEQFXtAvYB24Bp4A3g5g7mlSQtw8jBr6onGH6Pfu6YAm4ddS5J0sr5TVtJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGGHxJaoTBl6RGjBz8JBuTPJrkcJJDSW4fMmZLkuNJDg5+7hx1XknS8qzt4BwngS9X1YEk5wBPJ3mkqp6fN+7xqrq+g/kkSSsw8jv8qjpWVQcG268Dh4H1o55XktStTu/hJ7kI+DTwwyGHr0zyTJKHklza5bySpMWlqro5UfIh4N+Av6iqb887di7wi6o6kWQb8DdVtWmB80wBU4Pdi4EjnSxw5dYBP+15DePCa3GK1+IUr8Up43AtLqyqjww70Enwk0wA3wUerqqvLWH8i8BkVfV9YRaVZH9VTfa9jnHgtTjFa3GK1+KUcb8WXXxKJ8A3gMMLxT7J+YNxJNk8mPeVUeeWJC1dF5/SuQr4LPDjJAcHr30F+ARAVe0CbgBuSXISeBPYUV3dS5IkLcnIwa+qJ4AsMmYnsHPUuXqyu+8FjBGvxSlei1O8FqeM9bXo7Je2kqTx5qMVJKkRBn8BSbYmOZJkOskdfa+nT0n2JHk5yXN9r6VvS3mUSAuSnJXkR4Pv1hxK8ud9r6lvSdYk+c8k3+17LQsx+EMkWQPcDVwHXALcmOSSflfVq3uBrX0vYky8+yiRXwWuAG5t9J+Nt4BrqupTwGXA1iRX9Luk3t3O7JMGxpbBH24zMF1VL1TV28ADwPae19SbqnoMeLXvdYwDHyUyq2adGOxODH6a/YVgkg3A7wJ/2/daTsfgD7ceeGnO/gwN/kut01vkUSIfeINbGAeBl4FHqqrJ6zDw18AfA7/oeR2nZfCHG/Yx02bfvei9Bo8SeRD4UlW91vd6+lBVP6+qy4ANwOYkv9bzknqR5Hrg5ap6uu+1LMbgDzcDbJyzvwE42tNaNGYGjxJ5EPjW/OdGtaiq/hv4Ae3+nucq4PcGj4x5ALgmyd/1u6ThDP5wTwGbknwyyRnADmBvz2vSGFjKo0RakOQjSc4bbP8y8FvAT3pdVE+q6k+qakNVXcRsK/61qv6w52UNZfCHqKqTwG3Aw8z+Uu4fqupQv6vqT5L7gf8ALk4yk+SP+l5Tj959lMg1c/4Pbtv6XlQPLgAeTfIss2+QHqmqsf04omb5TVtJaoTv8CWpEQZfkhph8CWpEQZfkhph8CWpEQZfkhph8CWpEQZfkhrxf31GPiLnkeAOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example of label smoothing.\n",
    "crit = LabelSmoothing(5, 0, 0.4)\n",
    "predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],\n",
    "                             [0, 0.2, 0.7, 0.1, 0],\n",
    "                             [0, 0.2, 0.7, 0.1, 0]])\n",
    "v = crit(predict.log(),\n",
    "         torch.LongTensor([2, 1, 0]))\n",
    "\n",
    "# Show the target distributions expected by the system.\n",
    "plt.imshow(crit.true_dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "111b18b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0000, 0.1333, 0.6000, 0.1333, 0.1333],\n",
      "        [0.0000, 0.6000, 0.1333, 0.1333, 0.1333],\n",
      "        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])\n"
     ]
    }
   ],
   "source": [
    "print(crit.true_dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "390ca252",
   "metadata": {},
   "source": [
    "由于标签平滑的存在，如果模型对于某个单词特别有信心，输出特别大的概率，会被惩罚。如下代码所示，随着输入x的增大，x/d会越来越大，1/d会越来越小，但是loss并不是一直降低的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "21f1002f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x27221467490>]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZO0lEQVR4nO3deZAc53nf8e8zPddewAK7S+I+mEAkQVI8BJNyLDlUZIUA7RIsl6sMyY5sRTTDRHRkVaosKqqkKnZScWI7ZVdECkEYRowkm7FlRoZlSJTLEiXHEhkub4AgKBAggCVIYQHi2MUecz35o3t2ZwcLYADsYtDdv09xaqeP3XleHD+8fPt9u83dERGR+Mu0uwAREZkbCnQRkYRQoIuIJIQCXUQkIRToIiIJkW3XB/f39/uaNWva9fEiIrH07LPPHnX3gdmOtS3Q16xZw+DgYLs+XkQklszswNmOachFRCQhFOgiIgmhQBcRSQgFuohIQijQRUQSQoEuIpIQCnQRkYSIXaDveXuEP/j2Ho6NTra7FBGRK0rsAn3f8Cj/9Tt7OTKiQBcRaRS7QC/mAgAmytU2VyIicmWJbaCPK9BFRGaIYaCHJU+Wa22uRETkyhK7QO/Iq4cuIjKb2AV6MasxdBGR2cQu0NVDFxGZXewCfbqHrjF0EZFG8Qv0fFiyhlxERGaKXaDngwwZg/GSAl1EpFHsAt3MKOYC9dBFRJrELtABOnKBLoqKiDSJZaCHPXRdFBURaRTTQM9oyEVEpElMA11j6CIizWIZ6BpDFxE5UywDXT10EZEzxTbQx3VRVERkhpgGeoZJ9dBFRGaIZaBrDF1E5EzxDPS8Al1EpFksA10XRUVEzhTjQK/h7u0uRUTkihHTQI+eK1rRTBcRkbpYBnpHLnpqkW6hKyIyJZaBXowCfaKiQBcRqYtloKuHLiJyplgGen0MXbfQFRGZFtNAj3romrooIjKlpUA3s41mtsfM9prZA7McX2hmf2lmL5rZLjP7xNyXOm1qDF2BLiIy5byBbmYB8CCwCVgPfNTM1jed9ingFXe/GbgT+AMzy89xrVM6FOgiImdopYd+O7DX3fe5ewl4DNjcdI4DPWZmQDfwDlCZ00obdOQ15CIi0qyVQF8OHGrYHor2NfoCcD1wGHgZ+LS7n3HF0szuNbNBMxscHh6+yJKhmK330HVRVESkrpVAt1n2Na+5vwt4AVgG3AJ8wcwWnPFN7tvcfYO7bxgYGLjAUqcV82HZ6qGLiExrJdCHgJUN2ysIe+KNPgE87qG9wH7gurkp8Uz1i6K6J7qIyLRWAv0ZYJ2ZrY0udG4BtjedcxD4IICZXQ1cC+yby0IbaWGRiMiZsuc7wd0rZnY/8AQQAI+4+y4zuy86vhX4HeBLZvYy4RDNZ9396HwVnQsyBBnT0n8RkQbnDXQAd98B7Gjat7Xh/WHgH89taefWkQsYL+miqIhIXSxXikK4/F89dBGRaTEO9IAJjaGLiEyJdaBr2qKIyLTYBnqHnisqIjJDrANdPXQRkWmxDfRCLqOl/yIiDWIb6BpyERGZKbaBXlSgi4jMENtA1xi6iMhMsQ30osbQRURmiG+g59VDFxFpFN9AzwaUKjWqteZbs4uIpFNsA73+GLpJ3c9FRASIcaAXs9FTi3Q/FxERIMaBXu+hT1R0YVREBGIc6EU9tUhEZIbYB7oWF4mIhGIb6B0KdBGRGWIb6NM9dI2hi4hAjAO93kPX4iIRkVBsA72YC0vXkIuISCjGga4euohIo9gHunroIiKh2Ab61MIiBbqICBDjQJ9e+q9ZLiIiEONAzwYZcoExoZtziYgAMQ50CG+hq6X/IiKheAd6PtDtc0VEIrEO9I6ceugiInWxDnQ9V1REZFqsA70jp+eKiojUxTrQC7lA89BFRCItBbqZbTSzPWa218weOMs5d5rZC2a2y8y+N7dlzq5DgS4iMiV7vhPMLAAeBD4EDAHPmNl2d3+l4Zxe4CFgo7sfNLOr5qneGYq5jIZcREQirfTQbwf2uvs+dy8BjwGbm875GPC4ux8EcPcjc1vm7MIeui6KiohAa4G+HDjUsD0U7Wv0LmCRmT1pZs+a2cdn+0Fmdq+ZDZrZ4PDw8MVV3KCoi6IiIlNaCXSbZZ83bWeB9wA/C9wF/Bsze9cZ3+S+zd03uPuGgYGBCy62WVFj6CIiU847hk7YI1/ZsL0CODzLOUfd/TRw2sy+D9wMvDYnVZ6FAl1EZForPfRngHVmttbM8sAWYHvTOX8BvN/MsmbWCdwB7J7bUs/UkQsoV51KVePoIiLn7aG7e8XM7geeAALgEXffZWb3Rce3uvtuM/sW8BJQAx52953zWThARz56DF2lRncQ6yn1IiKXrJUhF9x9B7Cjad/Wpu3fA35v7ko7v8anFnUXWmqKiEhixbpbO/VcUd2gS0QkGYGuC6MiIjEP9I6pQNdFURGRWAd6MRc9V1Q9dBGReAd6Zz7soY+VKm2uRESk/WId6L2deQCOj5XaXImISPvFOtD7usJAPzaqQBcRiXWgLyjmCDLGO6cV6CIisQ70TMZY1JlXoIuIEPNAh3DY5ZgCXUQk/oG+uEs9dBERSEKgdyvQRUQgAYHe15Xn2Ohku8sQEWm72Af64q48pyYqlHVPdBFJudgHen0uuhYXiUjaxT7QF3cVADSOLiKpl4BAD3vo72i1qIikXOwDva87Wv6vHrqIpFzsA32qh65AF5GUi32g93bkAPXQRURiH+jZIENvZ453TmsuuoikW+wDHbT8X0QEEhLo4WpRBbqIpFsiAl09dBGRxAR6QYEuIqmXiEDv68pzfKxErebtLkVEpG0SEeiLu/LUHE6Ml9tdiohI2yQi0OurRTV1UUTSLBGBXl8tqpkuIpJmiQp0XRgVkTRLRKD3RbfQ1fJ/EUmzRAT6oq7wfi7qoYtImiUi0AvZgJ5CVoEuIqnWUqCb2UYz22Nme83sgXOc9xNmVjWzX5y7EluzuFurRUUk3c4b6GYWAA8Cm4D1wEfNbP1ZzvtPwBNzXWQrtPxfRNKulR767cBed9/n7iXgMWDzLOf9BvDnwJE5rK9lfV15XRQVkVRrJdCXA4catoeifVPMbDnwEWDruX6Qmd1rZoNmNjg8PHyhtZ5T2EPXwiIRSa9WAt1m2dd805Q/BD7r7tVz/SB33+buG9x9w8DAQIsltqZ+gy533c9FRNIp28I5Q8DKhu0VwOGmczYAj5kZQD9wt5lV3P3rc1FkK/q68pSrzshkhQXF3OX6WBGRK0Yrgf4MsM7M1gJvAluAjzWe4O5r6+/N7EvANy5nmEPDatHRkgJdRFLpvEMu7l4B7iecvbIb+FN332Vm95nZffNdYKsWRzfo0oVREUmrVnrouPsOYEfTvlkvgLr7r116WReuT/dzEZGUS8RKUYD+7vB+LsMjmukiIumUmEC/ekGRfJDhwLHT7S5FRKQtEhPoQcZYubiDNxToIpJSiQl0gLX9XbxxdKzdZYiItEWiAn11XxcH3jmth0WLSColKtDX9HcxUa7x45GJdpciInLZJSrQ1/Z1AWjYRURSKVGBvrqvE0AXRkUklRIV6Mt6O8gHGQW6iKRSogJ9auriUQW6iKRPogIdwqmLB45pDF1E0idxgb66r4s3jmnqooikT+ICvT518Yju6SIiKZO8QI9muuzXOLqIpEwCAz2ci66bdIlI2iQu0OtTF/cr0EUkZRIX6PWpiwe0WlREUiZxgQ7hsIsWF4lI2iQz0PvDQHfX1EURSY9kBnpfZ3jXxVOauigi6ZHMQO+P7rqoYRcRSZFkBvrUbXQV6CKSHokM9GW9HXTkAl59e6TdpYiIXDaJDPQgY7x7xUKeP3i83aWIiFw2iQx0gNtWL2LX4VNMlKvtLkVE5LJIbqCvWkSl5rz85sl2lyIiclkkNtBvXdULwHMHNOwiIumQ2EDv7y6wanEnz2kcXURSIrGBDnDbql6eO3hCK0ZFJBWSHeirFzE8MsnQ8fF2lyIiMu+SHeirFgHw/KET7S1EROQySHSgX7ekh45coAujIpIKLQW6mW00sz1mttfMHpjl+C+b2UvR6wdmdvPcl3rhskFGC4xEJDXOG+hmFgAPApuA9cBHzWx902n7gX/o7u8GfgfYNteFXiwtMBKRtGilh347sNfd97l7CXgM2Nx4grv/wN3r3eCngBVzW+bF0wIjEUmLVgJ9OXCoYXso2nc2nwS+OdsBM7vXzAbNbHB4eLj1Ki+BFhiJSFq0Eug2y75ZJ3ab2QcIA/2zsx13923uvsHdNwwMDLRe5SXo7y5wzUAXf/ujo5fl80RE2qWVQB8CVjZsrwAON59kZu8GHgY2u/uxuSlvbtx1wxJ+uO8YJ8ZK7S5FRGTetBLozwDrzGytmeWBLcD2xhPMbBXwOPBP3P21uS/z0my6cQnVmvPXr/y43aWIiMyb8wa6u1eA+4EngN3An7r7LjO7z8zui077t0Af8JCZvWBmg/NW8UW4aflClvd28K2db7e7FBGReZNt5SR33wHsaNq3teH9PcA9c1va3DEz7rphCV956gCjkxW6Cy01W0QkVhK9UrTRppuWUKrW+M6rR9pdiojIvEhNoN+2ahH93QWe0LCLiCRUagI9yBh33XA1391zRKtGRSSRUhPoAJtuXMpYqcr3Xrs8i5pERC6nVAX6HdcsZmFHjr966a12lyIiMudSFei5IMNHbl3ON3e+xY9PTbS7HBGROZWqQAf4pz+1lkrNefQHb7S7FBGROZW6QF/V18ld65fw1acPMlaqtLscEZE5k7pAB/j1n17LyfEyfzY41O5SRETmTCoD/bZVi7hlZS+P/N1+qrVZbxwpIhI7qQx0M+PX338NB46N6YZdIpIYqQx0gLtuuJoVizr44pN7qamXLiIJkNpAzwYZfvNn3sWLQyf52nMaSxeR+EttoAP8wq3L2bB6Eb/7zVc5OVZudzkiIpck1YGeyRi/vflGToyV+P1v72l3OSIilyTVgQ6wftkCPv6Ta/jK0wfY+ebJdpcjInLRUh/oAJ/50Lvo68rz+f/zMqVKrd3liIhcFAU6sLAjx29vvpEXh07yH/7qlXaXIyJyURTokbtvWso971vLoz88wOOa9SIiMaRAb/DAput47zWL+dzjL7PrsMbTRSReFOgNskGGL3zsNhZ15vlnX36WwyfG212SiEjLFOhN+rsLbPv4ezg5VmbLtqd4U6EuIjGhQJ/Fu1f08uV77uD4WIkt237I0PGxdpckInJeCvSzuGVlL1+95w5OjpX5pf/2FLvfOtXukkQkptydiXKVY6OTHHpnjKOjk/PyOebenhtTbdiwwQcHB9vy2Rdi55sn+eSjz3ByvMx//IWb+MitK9pdkojMs2rNOV2qcHqy/qqGX0v1r837Z3lfqjDWsK/xHoD//M6/x2c3XndRtZnZs+6+YbZj2Yv6iSly4/KF/OVvvI/7//h5PvO/X+T5gyf413dfTzEXtLs0EYm4O+PlKqOTFUYnwkAdrYdxqcLIxHQ4j06eGcqjDdujkxUmyq0vMOzMB3QVsnQXslPv+7vzrMp30lUIt2eek+X6pT3z8uugQG/BVT1FvnrPHfznb73Kf//b/XzvtWH+/c/fyPvXDbS7NJHYmgrhiTBE62Fcf396ssJIPYQnwiAenSxzerI6Y/9sPeBz6WoI165Clq5CwLLeYvR+Opjr4dtVCKbObQztrkKWzlxAJmPz+wt1ATTkcoF+sPcon//6TvYfPc3mW5bxWxuvY3lvR7vLErlsqjVvCuAyIxMzA/mM7RnBXGFkoszoZGshHGSM7qkAnhmuM94X6+8Dugu5sHecD/d3X6EBfDHONeSiQL8IE+UqDz35OluffB3H+aWfWMm/uPPvs0zBLlcwd+d0KewRj0yUGakH78R0wJ5q2q6Hc317ZKLCWKna0ud15YOpMO0u5uiJArmnmJsK46njs21H7wvZDGbxDuG5pECfJ0PHx3joydf5s8FDGMbP3byUX75jNbet6tUfQJlTlWotCtYKpxrCdWSiPP11al8UwBPTPeX697Ty170n6s32FMNXVyHLgnoIRyHbeKwe0D3Fhp5yPksQ857wlUqBPs+Gjo+x7fv7ePy5NxmdrHDdkh5+8T0r2HTTUg3HCKVKrSF4p8P31PjMffWQPhWde6ohlMfL5+8V57MZFhSzM3rAPdF2TzHLgmI9kHNTgdzTeH4xS3c+G/shiaRToF8mpycrbH/xMH/89EFeju6tfvPKXj50/VW8b90ANy1fqF5LzMwWxqcaesanGnrI04HccHy8zGQLt2SuX4RrDOAwhHNN+8LwXdCwrzs6t5DVzKs0UKC3wYFjp9nx8tt8c+dbvDQUhvvCjhy3r13Mrat6uXXlIm5asZDugiYazQd3Z6xUjYYc6iE8cyhiZHLmkMVo05DFqYlKS/fH78wHZ4RuvUfcE40dz348x4KOsCedDbTGT1pzyYFuZhuBPwIC4GF3/92m4xYdvxsYA37N3Z87189MeqA3OjY6yd+9foz/+6NhBt84zr6jp6eOrVzcwbVXL+DaJd2s6etibX8Xq/u66O/Op24cPlxNV5sxH7i+WKN5lkTjtLYZF/Emps893wwKM+jON48LT4fuguLsQdzYc1YYy+V2SQuLzCwAHgQ+BAwBz5jZdndvfBLEJmBd9LoD+GL0VYC+7gIfvnkZH755GQDHT5d4YegEu948ye63R9jz9gjf3XOEakMC5bMZli4ssmRBkYGeAgM9Bfq7CyzqzLOwI0dv58yLVB35gGI2IBfYvPxDUKs5pWqNyUqNUqXGZKXKZKXGRLnKRLnGZLnKRCV8P16qMl6uMlGuMlYKX+OlcHbEWLnKeLTabqx07tV051KfSxwOOeToLgQM9BRmHT+uD0n0FGYOU2i8WJKmlf/fvx3Y6+77AMzsMWAz0Bjom4H/5WF3/ykz6zWzpe7+1pxXnACLuvJ84Nqr+MC1V03tK1drDB0f541jpzl4bIzDJ8c5fGKCt0+Os+vwKYZHJhmdrJz3ZwcZIx9kyGcz5IIMucAIMtHLDML/MDNq7hD+R7XmVGuOu1OO3perNcrVGpWqU2k1aWeRC4yOXEBnPktnIaAzH75vXE1XX8AxteAjn21Y/BFMzajoKmgGhcjZtBLoy4FDDdtDnNn7nu2c5cCMQDeze4F7AVatWnWhtSZaLsiwtj8ccjmbiXKVk+NlToyVOTFWml7cMVlhvBT2iMfLVUqVGuWqM1mpUanWqLpPBzZEIe6Y2VS4BwaZjJExIxcY2UyGIGMUon8YsoGRz2bIBxkK2QyFXBB+jd4XswHFXIaOfEBHLnwVo/c5DUmIXBatBPpsXaHm7lor5+Du24BtEI6ht/DZ0qCYCyjmAq5eUGx3KSJyBWql6zQErGzYXgEcvohzRERkHrUS6M8A68xsrZnlgS3A9qZztgMft9B7gZMaPxcRubzOO+Ti7hUzux94gnDa4iPuvsvM7ouObwV2EE5Z3Es4bfET81eyiIjMpqVVLe6+gzC0G/dtbXjvwKfmtjQREbkQmn4gIpIQCnQRkYRQoIuIJIQCXUQkIdp2t0UzGwYOXMC39ANH56mcK1la2w3pbbvanS4X2u7V7j7rA43bFugXyswGz3aHsSRLa7shvW1Xu9NlLtutIRcRkYRQoIuIJEScAn1buwtok7S2G9LbdrU7Xeas3bEZQxcRkXOLUw9dRETOQYEuIpIQsQh0M9toZnvMbK+ZPdDueuaLma00s++a2W4z22Vmn472LzazvzazH0VfF7W71vlgZoGZPW9m34i2E9/u6HGNXzOzV6Pf959MSbs/E/0Z32lmf2JmxSS228weMbMjZrazYd9Z22lmn4tybo+Z3XWhn3fFB3rDQ6o3AeuBj5rZ+vZWNW8qwL9y9+uB9wKfitr6APA37r4O+JtoO4k+Dexu2E5Du/8I+Ja7XwfcTNj+RLfbzJYD/xLY4O43Et6WewvJbPeXgI1N+2ZtZ/R3fQtwQ/Q9D0X517IrPtBpeEi1u5eA+kOqE8fd33L356L3I4R/uZcTtvfR6LRHgZ9vS4HzyMxWAD8LPNywO9HtNrMFwE8D/wPA3UvufoKEtzuSBTrMLAt0Ej7hLHHtdvfvA+807T5bOzcDj7n7pLvvJ3y+xO0X8nlxCPSzPYA60cxsDXAr8DRwdf0JUNHXq9pY2nz5Q+C3gFrDvqS3+xpgGPif0VDTw2bWRcLb7e5vAr8PHCR8kPxJd/82CW93g7O185KzLg6B3tIDqJPEzLqBPwd+091Ptbue+WZmPwcccfdn213LZZYFbgO+6O63AqdJxjDDOUVjxpuBtcAyoMvMfqW9VV0RLjnr4hDoqXoAtZnlCMP8q+7+eLT7x2a2NDq+FDjSrvrmyU8BHzazNwiH1P6RmX2F5Ld7CBhy96ej7a8RBnzS2/0zwH53H3b3MvA48A9IfrvrztbOS866OAR6Kw+pTgQzM8Lx1N3u/l8aDm0HfjV6/6vAX1zu2uaTu3/O3Ve4+xrC39/vuPuvkPx2vw0cMrNro10fBF4h4e0mHGp5r5l1Rn/mP0h4vSjp7a47Wzu3A1vMrGBma4F1wP+7oJ/s7lf8i/AB1K8BrwOfb3c989jO9xH+L9ZLwAvR626gj/Bq+I+ir4vbXes8/hrcCXwjep/4dgO3AIPR7/nXgUUpafe/A14FdgJfBgpJbDfwJ4TXCcqEPfBPnqudwOejnNsDbLrQz9PSfxGRhIjDkIuIiLRAgS4ikhAKdBGRhFCgi4gkhAJdRCQhFOgiIgmhQBcRSYj/D2Ll3Fs1UbBhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "crit = LabelSmoothing(5, 0, 0.1)\n",
    "\n",
    "\n",
    "def loss(x):\n",
    "    d = x + 3 * 1\n",
    "    predict = torch.FloatTensor([[0, x / d, 1 / d, 1 / d, 1 / d]])\n",
    "    # print(predict)\n",
    "    return crit(predict.log(),\n",
    "                torch.LongTensor([1])).item()\n",
    "\n",
    "\n",
    "y = [loss(x) for x in range(1, 100)]\n",
    "x = np.arange(1, 100)\n",
    "plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e2d9d4",
   "metadata": {},
   "source": [
    "## 实例\n",
    "### 合成数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "3f2fc12c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_gen(V, batch, nbatches):\n",
    "    \"Generate random data for a src-tgt copy task.\"\n",
    "    for i in range(nbatches):\n",
    "        data = torch.from_numpy(np.random.randint(1, V, size=(batch, 10)))\n",
    "        data[:, 0] = 1\n",
    "        src = data.long().requires_grad_(False)\n",
    "        tgt = data.long().requires_grad_(False)\n",
    "        yield Batch(src, tgt, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab556f75",
   "metadata": {},
   "source": [
    "### 损失函数计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2f8e28b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleLossCompute:\n",
    "    \"A simple loss compute and train function.\"\n",
    "\n",
    "    def __init__(self, generator, criterion, opt=None):\n",
    "        self.generator = generator\n",
    "        self.criterion = criterion\n",
    "        self.opt = opt\n",
    "\n",
    "    def __call__(self, x, y, norm):\n",
    "        x = self.generator(x)\n",
    "        loss = self.criterion(x.contiguous().view(-1, x.size(-1)),\n",
    "                              y.contiguous().view(-1)) / norm\n",
    "        loss.backward()\n",
    "        if self.opt is not None:\n",
    "            self.opt.step()\n",
    "            self.opt.optimizer.zero_grad()\n",
    "        return loss.item() * norm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ecfab8",
   "metadata": {},
   "source": [
    "### 贪婪解码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "9baac41b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch Step: 1 Loss: 3.217873 Tokens per Sec: 413.666809\n",
      "Epoch Step: 1 Loss: 2.044819 Tokens per Sec: 567.102539\n",
      "tensor(2.0707)\n",
      "Epoch Step: 1 Loss: 2.084461 Tokens per Sec: 378.063538\n",
      "Epoch Step: 1 Loss: 1.725667 Tokens per Sec: 655.580994\n",
      "tensor(1.7515)\n",
      "Epoch Step: 1 Loss: 2.133247 Tokens per Sec: 411.176636\n",
      "Epoch Step: 1 Loss: 1.681059 Tokens per Sec: 684.259460\n",
      "tensor(1.6964)\n",
      "Epoch Step: 1 Loss: 1.905264 Tokens per Sec: 442.886932\n",
      "Epoch Step: 1 Loss: 1.337842 Tokens per Sec: 659.193665\n",
      "tensor(1.3322)\n",
      "Epoch Step: 1 Loss: 1.527641 Tokens per Sec: 457.912903\n",
      "Epoch Step: 1 Loss: 1.006559 Tokens per Sec: 614.896240\n",
      "tensor(0.9990)\n",
      "Epoch Step: 1 Loss: 1.470397 Tokens per Sec: 460.250824\n",
      "Epoch Step: 1 Loss: 0.759082 Tokens per Sec: 687.745728\n",
      "tensor(0.7827)\n",
      "Epoch Step: 1 Loss: 1.000888 Tokens per Sec: 470.682098\n",
      "Epoch Step: 1 Loss: 0.634258 Tokens per Sec: 603.893799\n",
      "tensor(0.6654)\n",
      "Epoch Step: 1 Loss: 0.799201 Tokens per Sec: 457.523834\n",
      "Epoch Step: 1 Loss: 0.370392 Tokens per Sec: 654.398865\n",
      "tensor(0.3792)\n",
      "Epoch Step: 1 Loss: 0.777453 Tokens per Sec: 454.825775\n",
      "Epoch Step: 1 Loss: 0.390272 Tokens per Sec: 615.598022\n",
      "tensor(0.3837)\n",
      "Epoch Step: 1 Loss: 0.712812 Tokens per Sec: 454.242004\n",
      "Epoch Step: 1 Loss: 0.302385 Tokens per Sec: 692.153076\n",
      "tensor(0.3265)\n"
     ]
    }
   ],
   "source": [
    "# Train the simple copy task.\n",
    "V = 11\n",
    "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
    "model = make_model(V, V, N=2)\n",
    "model_opt = NoamOpt(model.src_embed[0].d_model, 1, 400,\n",
    "        torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))\n",
    "\n",
    "for epoch in range(10):\n",
    "    model.train()\n",
    "    run_epoch(data_gen(V, 30, 20), model, \n",
    "              SimpleLossCompute(model.generator, criterion, model_opt))\n",
    "    model.eval()\n",
    "    print(run_epoch(data_gen(V, 30, 5), model, \n",
    "                    SimpleLossCompute(model.generator, criterion, None)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "08a674d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 2, 3, 5, 6, 7, 8, 9, 8, 9]])\n"
     ]
    }
   ],
   "source": [
    "def greedy_decode(model, src, src_mask, max_len, start_symbol):\n",
    "    memory = model.encode(src, src_mask)\n",
    "    ys = torch.ones(1, 1).fill_(start_symbol).type_as(src.data)\n",
    "    for i in range(max_len-1):\n",
    "        out = model.decode(memory, src_mask,\n",
    "                           ys,\n",
    "                           subsequent_mask(ys.size(1)).type_as(src.data))\n",
    "        prob = model.generator(out[:, -1])\n",
    "        _, next_word = torch.max(prob, dim=1)\n",
    "        next_word = next_word.data[0]\n",
    "        ys = torch.cat([ys,\n",
    "                        torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=1)\n",
    "    return ys\n",
    "\n",
    "\n",
    "model.eval()\n",
    "src = torch.LongTensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]])\n",
    "src_mask = torch.ones(1, 1, 10)\n",
    "print(greedy_decode(model, src, src_mask, max_len=10, start_symbol=1))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
